[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:26:42 dist-test-slave-9wb0 krb5kdc[1187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:26:42 dist-test-slave-9wb0 krb5kdc[1187](info): set up 2 sockets
Sep 26 19:26:42 dist-test-slave-9wb0 krb5kdc[1187](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 19:26:43.108157 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 19:26:43.249480 1180 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.141s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 19:26:43.491647 1180 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.242s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 19:26:43.783507 1180 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.292s user 0.000s sys 0.004s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/test-user.keytab.
W20260926 19:26:44.748665 1180 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.965s user 0.000s sys 0.004s
Sep 26 19:26:44 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
W20260926 19:26:45.941872 1180 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.39.62: real 1.149s user 0.000s sys 0.008s
I20260926 19:26:45.943579 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41835
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:41835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:46.299796 1200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:46.300165 1200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:46.300392 1200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:46.311262 1200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:26:46.311420 1200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:46.311525 1200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:46.311633 1200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:26:46.311718 1200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:26:46.322832 1200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:41835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:46.324275 1200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:46.325675 1200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:46.338833 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:46.338883 1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:46.339001 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:46.339792 1200 server_base.cc:1034] running on GCE node
I20260926 19:26:46.340648 1200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:46.342286 1200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:46.343597 1200 hybrid_clock.cc:648] HybridClock initialized: now 1790450806343489 us; error 105 us; skew 500 ppm
I20260926 19:26:46.343981 1200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:46 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450806, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:46.353721 1200 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:26:46.355571 1200 webserver.cc:460] Webserver started at http://127.1.39.62:45659/ using document root <none> and password file <none>
I20260926 19:26:46.356340 1200 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:46.356531 1200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:46.356865 1200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:46.360419 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "d2a82eb35bab48c587493dd6470c994a"
format_stamp: "Formatted at 2026-09-26 19:26:46 on dist-test-slave-9wb0"
server_key: "21387d15144e7b8f42b08096e63e5bad"
server_key_iv: "dd53ed6473e1218f5b9b2f33d4efe7fb"
server_key_version: "encryptionkey@0"
I20260926 19:26:46.361261 1200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "d2a82eb35bab48c587493dd6470c994a"
format_stamp: "Formatted at 2026-09-26 19:26:46 on dist-test-slave-9wb0"
server_key: "21387d15144e7b8f42b08096e63e5bad"
server_key_iv: "dd53ed6473e1218f5b9b2f33d4efe7fb"
server_key_version: "encryptionkey@0"
I20260926 19:26:46.367516 1200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 19:26:46.371908 1215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:46.373543 1200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 19:26:46.373781 1200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "d2a82eb35bab48c587493dd6470c994a"
format_stamp: "Formatted at 2026-09-26 19:26:46 on dist-test-slave-9wb0"
server_key: "21387d15144e7b8f42b08096e63e5bad"
server_key_iv: "dd53ed6473e1218f5b9b2f33d4efe7fb"
server_key_version: "encryptionkey@0"
I20260926 19:26:46.373988 1200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:46.396123 1200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:46.401059 1200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:46.401436 1200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:46.424000 1200 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:41835
I20260926 19:26:46.424019 1266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:41835 every 8 connection(s)
I20260926 19:26:46.425585 1200 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:26:46.429556 1267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:46.431762 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1200
I20260926 19:26:46.432032 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:26:46.434077 1180 external_mini_cluster.cc:1428] Setting key 0b12573f3e6451a5689aaabccc147187
I20260926 19:26:46.441749 1267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a: Bootstrap starting.
I20260926 19:26:46.444900 1267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:46.446061 1267 log.cc:824] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:46.448988 1267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a: No bootstrap required, opened a new log
I20260926 19:26:46.457170 1267 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } }
I20260926 19:26:46.457655 1267 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:46.457774 1267 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a82eb35bab48c587493dd6470c994a, State: Initialized, Role: FOLLOWER
I20260926 19:26:46.458424 1267 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } }
I20260926 19:26:46.458683 1267 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:26:46.458837 1267 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:26:46.459028 1267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 0 FOLLOWER]: Advancing to term 1
Sep 26 19:26:46 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:46.474085 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:46.438143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.1.39.62:41835)
0926 19:26:46.439111 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:46.439133 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:46.440174 (+ 1041us) server_negotiation.cc:407] Connection header received
0926 19:26:46.450345 (+ 10171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:46.450371 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:46.450833 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:46.451274 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:46.454451 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:46.456357 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:46.458503 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:46.459400 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:46.459616 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:46.466435 (+ 6819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:46.466444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:46.466460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:46.466502 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:46.469451 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:46.470279 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:46.470286 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:46.470292 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:46.470460 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:46.470868 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:46.470871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:46.470873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:46.471507 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:46.471745 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:46.472575 (+ 830us) server_negotiation.cc:299] Negotiation successful
0926 19:26:46.472932 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":427,"threads_started":1}
I20260926 19:26:46.511322 1267 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } }
I20260926 19:26:46.512121 1267 leader_election.cc:302] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2a82eb35bab48c587493dd6470c994a; no voters:
I20260926 19:26:46.513057 1267 leader_election.cc:288] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:26:46.513190 1272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:46.513935 1272 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [term 1 LEADER]: Becoming Leader. State: Replica: d2a82eb35bab48c587493dd6470c994a, State: Running, Role: LEADER
I20260926 19:26:46.514604 1272 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } }
I20260926 19:26:46.516991 1267 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:26:46.518838 1273 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2a82eb35bab48c587493dd6470c994a. Latest consensus state: current_term: 1 leader_uuid: "d2a82eb35bab48c587493dd6470c994a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } } }
I20260926 19:26:46.518817 1274 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2a82eb35bab48c587493dd6470c994a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2a82eb35bab48c587493dd6470c994a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41835 } } }
I20260926 19:26:46.519345 1274 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:46.520205 1273 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:46.521808 1278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:26:46.528229 1278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:26:46.538300 1278 catalog_manager.cc:1347] Generated new cluster ID: 096ae5b3ce1444e585aa85f1d2f3a719
I20260926 19:26:46.538478 1278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:26:46.550099 1278 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:26:46.552023 1278 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:26:46.561627 1278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2a82eb35bab48c587493dd6470c994a: Generated new TSK 0
I20260926 19:26:46.562712 1278 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
W20260926 19:26:47.316408 1180 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.39.1: real 0.734s user 0.001s sys 0.007s
I20260926 19:26:47.317498 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:47.662709 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:47.663025 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:47.663203 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:47.670827 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:47.671423 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:47.671635 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:26:47.681969 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:47.683382 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:47.684841 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:47.698652 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:47.698693 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:47.698728 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:47.699512 1293 server_base.cc:1034] running on GCE node
I20260926 19:26:47.700048 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:47.700923 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:47.702523 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790450807702065 us; error 55 us; skew 500 ppm
I20260926 19:26:47.702859 1293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:47 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450807, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:47.706020 1293 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:26:47.707612 1293 webserver.cc:460] Webserver started at http://127.1.39.1:41599/ using document root <none> and password file <none>
I20260926 19:26:47.708290 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:47.708424 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:47.708717 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:47.711872 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "c578905f83dc400a899e08d937cfec47"
format_stamp: "Formatted at 2026-09-26 19:26:47 on dist-test-slave-9wb0"
server_key: "4a2efcbc349c741accc8e3c54669cf58"
server_key_iv: "77d434e06c4335bd4c8769758e94ed08"
server_key_version: "encryptionkey@0"
I20260926 19:26:47.712647 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "c578905f83dc400a899e08d937cfec47"
format_stamp: "Formatted at 2026-09-26 19:26:47 on dist-test-slave-9wb0"
server_key: "4a2efcbc349c741accc8e3c54669cf58"
server_key_iv: "77d434e06c4335bd4c8769758e94ed08"
server_key_version: "encryptionkey@0"
I20260926 19:26:47.718529 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260926 19:26:47.722092 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:47.723506 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:26:47.723673 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "c578905f83dc400a899e08d937cfec47"
format_stamp: "Formatted at 2026-09-26 19:26:47 on dist-test-slave-9wb0"
server_key: "4a2efcbc349c741accc8e3c54669cf58"
server_key_iv: "77d434e06c4335bd4c8769758e94ed08"
server_key_version: "encryptionkey@0"
I20260926 19:26:47.723869 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:47.751442 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:47.755887 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:47.756186 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:47.757143 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:47.758512 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:47.758618 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:47.758765 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:47.758837 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:47.796712 1293 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:45003
I20260926 19:26:47.796734 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:45003 every 8 connection(s)
I20260926 19:26:47.798110 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:26:47.804847 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1293
I20260926 19:26:47.805084 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:26:47.805759 1180 external_mini_cluster.cc:1428] Setting key 6004d6961eb65e30e6e2c9ef6c43e572
Sep 26 19:26:47 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450807, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:47.818413 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:47.800718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:40167 (local address 127.1.39.62:41835)
0926 19:26:47.801181 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:47.801185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:47.802055 (+ 870us) server_negotiation.cc:407] Connection header received
0926 19:26:47.803937 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:47.803943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:47.804039 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:47.804163 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:47.805827 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:47.806584 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:47.808125 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:47.808567 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:47.808705 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:47.812123 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:47.812133 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:47.812136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:47.812165 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:47.815287 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:47.816014 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:47.816018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:47.816021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:47.816132 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:47.816595 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:47.816600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:47.816604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:47.816854 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:47.817022 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:47.817829 (+ 807us) server_negotiation.cc:299] Negotiation successful
0926 19:26:47.818046 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":241,"threads_started":1}
I20260926 19:26:47.819423 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:47.801157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41835 (local address 127.1.39.1:40167)
0926 19:26:47.801882 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:47.801926 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:47.803669 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:47.804409 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:47.804416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:47.804946 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:47.805645 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:47.805663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:47.806748 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:47.806754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:47.807979 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:47.807990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:47.808791 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:47.808798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:47.809041 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:47.809746 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:47.809768 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:47.811903 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:47.815420 (+ 3517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:47.815429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:47.815445 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:47.815877 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:47.816241 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:47.816247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:47.816250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:47.816480 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:47.816989 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:47.816997 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:47.817400 (+ 403us) client_negotiation.cc:769] Sending connection context
0926 19:26:47.817733 (+ 333us) client_negotiation.cc:240] Negotiation successful
0926 19:26:47.818246 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":334,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:26:47.820978 1421 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41835
I20260926 19:26:47.821344 1421 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:47.822126 1421 heartbeater.cc:511] Master 127.1.39.62:41835 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
I20260926 19:26:47.824575 1232 ts_manager.cc:194] Registered new tserver with Master: c578905f83dc400a899e08d937cfec47 (127.1.39.1:45003)
I20260926 19:26:47.827201 1232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:40167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:26:47.846410 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:48.187350 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:48.187683 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:48.187796 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:48.195070 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:48.195175 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:48.195289 1427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:26:48.205698 1427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:48.206977 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:48.208235 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:48.221980 1432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:48.222332 1433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:48.222616 1435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:48.223790 1427 server_base.cc:1034] running on GCE node
I20260926 19:26:48.224450 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:48.225446 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:48.226675 1427 hybrid_clock.cc:648] HybridClock initialized: now 1790450808226621 us; error 49 us; skew 500 ppm
I20260926 19:26:48.227011 1427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:48.230391 1427 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:26:48.231981 1427 webserver.cc:460] Webserver started at http://127.1.39.2:37893/ using document root <none> and password file <none>
I20260926 19:26:48.232764 1427 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:48.232903 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:48.233214 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:48.236219 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "f576a26388cb484b8b675e50fddd5b76"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "03551bdd480cc3a507f282d698375327"
server_key_iv: "4b8e51912685f0b82d7e2959cd3c6442"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.237048 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "f576a26388cb484b8b675e50fddd5b76"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "03551bdd480cc3a507f282d698375327"
server_key_iv: "4b8e51912685f0b82d7e2959cd3c6442"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.242050 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 19:26:48.246039 1442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.247442 1427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:26:48.247641 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "f576a26388cb484b8b675e50fddd5b76"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "03551bdd480cc3a507f282d698375327"
server_key_iv: "4b8e51912685f0b82d7e2959cd3c6442"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.247843 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:48.272590 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:48.276435 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:48.276767 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:48.277755 1427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:48.279088 1427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:48.279187 1427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.279333 1427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:48.279408 1427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.317693 1427 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:41499
I20260926 19:26:48.317721 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:41499 every 8 connection(s)
I20260926 19:26:48.319147 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:26:48.324357 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1427
I20260926 19:26:48.324563 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:26:48.325042 1180 external_mini_cluster.cc:1428] Setting key 297f31f76226e98f2dd8a8fcb21d790d
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:48.341714 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.321642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:42753 (local address 127.1.39.62:41835)
0926 19:26:48.322071 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:48.322077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:48.323083 (+ 1006us) server_negotiation.cc:407] Connection header received
0926 19:26:48.324946 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:48.324952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:48.325031 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:48.325156 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:48.327175 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.328402 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.330114 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.330854 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.330997 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.335777 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:48.335782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:48.335785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:48.335815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:48.338410 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.339352 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.339356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.339358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.339483 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.339994 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.339999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.340001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.340238 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:48.340341 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:48.341242 (+ 901us) server_negotiation.cc:299] Negotiation successful
0926 19:26:48.341450 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":195,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:26:48.343266 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.322175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41835 (local address 127.1.39.2:42753)
0926 19:26:48.322943 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:48.322981 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:48.324675 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:48.325522 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:48.325530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:48.326219 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:48.327002 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:48.327027 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.328560 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.328564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:48.329973 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:48.329984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.331078 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.331083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:48.331277 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.332044 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:48.332078 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:48.335560 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:48.338599 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:48.338612 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:48.338633 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:48.339231 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:48.339619 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:48.339627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:48.339632 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:48.339880 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:48.340373 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:48.340385 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:48.340848 (+ 463us) client_negotiation.cc:769] Sending connection context
0926 19:26:48.341159 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 19:26:48.341763 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":341,"threads_started":1}
I20260926 19:26:48.344879 1555 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41835
I20260926 19:26:48.345198 1555 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
I20260926 19:26:48.346041 1555 heartbeater.cc:511] Master 127.1.39.62:41835 requested a full tablet report, sending...
I20260926 19:26:48.347549 1232 ts_manager.cc:194] Registered new tserver with Master: f576a26388cb484b8b675e50fddd5b76 (127.1.39.2:41499)
I20260926 19:26:48.348819 1232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:42753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:26:48.369270 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:48.715341 1561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:48.715685 1561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:48.715850 1561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:48.723357 1561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:48.723498 1561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:48.723659 1561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:26:48.733506 1561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41835
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:48.734781 1561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:48.736624 1561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:48.749607 1566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:48.749565 1569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:48.749647 1567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:48.750562 1561 server_base.cc:1034] running on GCE node
I20260926 19:26:48.751158 1561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:48.752113 1561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:48.753350 1561 hybrid_clock.cc:648] HybridClock initialized: now 1790450808753260 us; error 38 us; skew 500 ppm
I20260926 19:26:48.753682 1561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:48.757174 1561 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:26:48.758867 1561 webserver.cc:460] Webserver started at http://127.1.39.3:33493/ using document root <none> and password file <none>
I20260926 19:26:48.759573 1561 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:48.759675 1561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:48.759922 1561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:48.762975 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "c25a020edaef4f989c9a957cacabaf15"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "808383a29bb6d92c9484ecaa98f90885"
server_key_iv: "a0435eb693e5919a2f1c4b298d43c666"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.763711 1561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "c25a020edaef4f989c9a957cacabaf15"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "808383a29bb6d92c9484ecaa98f90885"
server_key_iv: "a0435eb693e5919a2f1c4b298d43c666"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.769439 1561 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 19:26:48.772941 1576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.774415 1561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:26:48.774597 1561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "c25a020edaef4f989c9a957cacabaf15"
format_stamp: "Formatted at 2026-09-26 19:26:48 on dist-test-slave-9wb0"
server_key: "808383a29bb6d92c9484ecaa98f90885"
server_key_iv: "a0435eb693e5919a2f1c4b298d43c666"
server_key_version: "encryptionkey@0"
I20260926 19:26:48.774797 1561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:48.787935 1561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:48.792263 1561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:48.792585 1561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:48.793601 1561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:48.794869 1561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:48.795130 1561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.795295 1561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:48.795380 1561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:48.831135 1421 heartbeater.cc:503] Master 127.1.39.62:41835 was elected leader, sending a full tablet report...
I20260926 19:26:48.834563 1561 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38543
I20260926 19:26:48.834609 1688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38543 every 8 connection(s)
I20260926 19:26:48.836104 1561 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:26:48.846349 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1561
I20260926 19:26:48.846609 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:26:48.847147 1180 external_mini_cluster.cc:1428] Setting key aaa9a988b19cf306beaec680b2d322af
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:48.858878 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.839085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:33131 (local address 127.1.39.62:41835)
0926 19:26:48.839261 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:48.839266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:48.841392 (+ 2126us) server_negotiation.cc:407] Connection header received
0926 19:26:48.843473 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:48.843486 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:48.843628 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:48.843758 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:48.845511 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.846268 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.848352 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.848827 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.848979 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.852911 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:48.852916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:48.852919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:48.852950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:48.855786 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.856555 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.856568 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.856570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.856683 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.857121 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.857126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.857129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.857475 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:48.857625 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:48.858322 (+ 697us) server_negotiation.cc:299] Negotiation successful
0926 19:26:48.858559 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:26:48.859875 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.839515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41835 (local address 127.1.39.3:33131)
0926 19:26:48.841099 (+ 1584us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:48.841151 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:48.842954 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:48.844018 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:48.844029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:48.844584 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:48.845262 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:48.845282 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.846431 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.846437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:48.848093 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:48.848111 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.849069 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.849076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:48.849401 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.850103 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:48.850140 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:48.852696 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:48.855953 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:48.855962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:48.855980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:48.856424 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:48.856797 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:48.856802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:48.856806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:48.857001 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:48.857599 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:48.857606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:48.857952 (+ 346us) client_negotiation.cc:769] Sending connection context
0926 19:26:48.858228 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 19:26:48.858777 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":663,"threads_started":1}
I20260926 19:26:48.861557 1689 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41835
I20260926 19:26:48.861972 1689 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:48.863029 1689 heartbeater.cc:511] Master 127.1.39.62:41835 requested a full tablet report, sending...
I20260926 19:26:48.864760 1232 ts_manager.cc:194] Registered new tserver with Master: c25a020edaef4f989c9a957cacabaf15 (127.1.39.3:38543)
I20260926 19:26:48.865793 1232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:33131
I20260926 19:26:48.876140 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:48.923383 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.906383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.1.39.1:45003)
0926 19:26:48.906974 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:48.906982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:48.907013 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:26:48.907074 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:48.907081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:48.907311 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:48.907441 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:48.909029 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.910138 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.911550 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.912135 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.912243 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.916294 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:48.916306 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:48.916321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:48.916358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:48.920365 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.921094 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.921102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.921104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.921239 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.921789 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.921793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.921795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.922211 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:48.922475 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:48.922747 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 19:26:48.923012 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":292,"threads_started":1}
W20260926 19:26:48.924737 1336 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38752
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:26:48 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:48.970153 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.955415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.1.39.1:45003)
0926 19:26:48.955626 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:48.955633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:48.955982 (+ 349us) server_negotiation.cc:407] Connection header received
0926 19:26:48.956205 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:48.956209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:48.956303 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:48.956413 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:48.957822 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.958514 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.959795 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.960207 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.960333 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.964157 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:48.964162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:48.964166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:48.964198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:48.967308 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.968069 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.968075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.968079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.968238 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.968708 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.968751 (+ 43us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.968756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.969178 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:48.969454 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:48.969738 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 19:26:48.969912 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:26:48.971648 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1293
I20260926 19:26:48.984982 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:48.972855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38772 (local address 127.1.39.1:45003)
0926 19:26:48.973090 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:48.973095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:48.973354 (+ 259us) server_negotiation.cc:407] Connection header received
0926 19:26:48.973570 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:48.973577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:48.973636 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:48.973733 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:48.974849 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.975539 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.976807 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:48.977219 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:48.977395 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:48.979992 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:48.979996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:48.980000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:48.980028 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:48.982507 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.983121 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.983128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.983132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.983267 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:48.983643 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:48.983649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:48.983652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:48.983897 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:48.984135 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:48.984415 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 19:26:48.984615 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 19:26:48.985860 1416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:49.068135 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1293
I20260926 19:26:49.085103 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1427
Sep 26 19:26:49 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:26:49.102301 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:49.086285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.1.39.2:41499)
0926 19:26:49.086953 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:49.086961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:49.086987 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:26:49.087051 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:49.087058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:49.087390 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:49.087573 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:49.088867 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:49.090484 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:49.091732 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:49.092356 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:49.092508 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:49.096656 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:49.096669 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:49.096685 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:49.096722 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:49.099594 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:49.100249 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:49.100256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:49.100258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:49.100379 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:49.100855 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:49.100859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:49.100861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:49.101185 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:49.101493 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:49.101720 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 19:26:49.101909 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":310,"threads_started":1}
I20260926 19:26:49.103612 1550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:49.186791 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1427
I20260926 19:26:49.209575 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1561
Sep 26 19:26:49 dist-test-slave-9wb0 krb5kdc[1187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:26:49.227322 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:49.210748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.1.39.3:38543)
0926 19:26:49.211513 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:49.211520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:49.211553 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:26:49.211616 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:49.211623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:49.211835 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:49.211986 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:49.213216 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:49.214618 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:49.216033 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:49.216622 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:49.216729 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:49.221415 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:49.221432 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:49.221455 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:49.221524 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:49.224560 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:49.225239 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:49.225246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:49.225249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:49.225459 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:49.225865 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:49.225869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:49.225871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:49.226254 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:49.226498 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:49.226821 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 19:26:49.227040 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":428,"threads_started":1}
I20260926 19:26:49.228672 1684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:49.310458 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1561
I20260926 19:26:49.327026 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1200
I20260926 19:26:49.328300 1262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:49.429621 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1200
2026-09-26T19:26:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (7464 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:26:49 dist-test-slave-9wb0 krb5kdc[1710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:26:49 dist-test-slave-9wb0 krb5kdc[1710](info): set up 2 sockets
Sep 26 19:26:49 dist-test-slave-9wb0 krb5kdc[1710](info): commencing operation
krb5kdc: starting...
W20260926 19:26:50.517247 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:26:50 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:26:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:26:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:26:50.693027 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:39175
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:39175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:51.013417 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:51.013780 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:51.013942 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:51.021247 1723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:26:51.021432 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:51.021539 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:51.021637 1723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:26:51.021725 1723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:26:51.031798 1723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:39175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:39175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:51.033185 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:51.034582 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:51.046116 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:51.046154 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:51.046353 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:51.047365 1723 server_base.cc:1034] running on GCE node
I20260926 19:26:51.048261 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:51.050038 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:51.051316 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790450811051263 us; error 61 us; skew 500 ppm
I20260926 19:26:51.051710 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:51 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450811, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:51.054926 1723 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:26:51.056349 1723 webserver.cc:460] Webserver started at http://127.1.39.62:46589/ using document root <none> and password file <none>
I20260926 19:26:51.057015 1723 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:51.057116 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:51.057461 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:51.060436 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "7f48c246f143422282b2f669abfe33de"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "87b6a9bafdcf5bd1a7b27bdac96c4429"
server_key_iv: "46ab2dd458fe5d6387db67f57576f760"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.061137 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "7f48c246f143422282b2f669abfe33de"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "87b6a9bafdcf5bd1a7b27bdac96c4429"
server_key_iv: "46ab2dd458fe5d6387db67f57576f760"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.066546 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 19:26:51.070210 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:51.071641 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:26:51.071787 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "7f48c246f143422282b2f669abfe33de"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "87b6a9bafdcf5bd1a7b27bdac96c4429"
server_key_iv: "46ab2dd458fe5d6387db67f57576f760"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.071959 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:51.102208 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:51.106683 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:51.106925 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:51.125386 1723 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:39175
I20260926 19:26:51.125501 1789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:39175 every 8 connection(s)
I20260926 19:26:51.126820 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:26:51.129631 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1723
I20260926 19:26:51.129835 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:26:51.130499 1180 external_mini_cluster.cc:1428] Setting key ad9c8390d7e571fb8d9851f0e3466e03
I20260926 19:26:51.130661 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:51.145234 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de: Bootstrap starting.
Sep 26 19:26:51 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:51.148509 1790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:51.149645 1790 log.cc:824] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:51.152266 1790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de: No bootstrap required, opened a new log
I20260926 19:26:51.154047 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:51.132695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40300 (local address 127.1.39.62:39175)
0926 19:26:51.133595 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:51.133615 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:51.133671 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:26:51.135481 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:51.135507 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:51.135986 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:51.136416 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:51.137845 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.139799 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.141620 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.142394 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.142558 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:51.147080 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:51.147089 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:51.147102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:51.147150 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:51.150189 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:51.150843 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:51.150849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:51.150854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:51.151016 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:51.151398 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:51.151407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:51.151409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:51.151870 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:51.152161 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:51.152648 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 19:26:51.153041 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":391,"threads_started":1}
I20260926 19:26:51.160017 1790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } }
I20260926 19:26:51.160475 1790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:51.160581 1790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f48c246f143422282b2f669abfe33de, State: Initialized, Role: FOLLOWER
I20260926 19:26:51.161182 1790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } }
I20260926 19:26:51.161463 1790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:26:51.161593 1790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:26:51.161768 1790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:51.163806 1790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } }
I20260926 19:26:51.164312 1790 leader_election.cc:302] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f48c246f143422282b2f669abfe33de; no voters:
I20260926 19:26:51.164852 1790 leader_election.cc:288] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:26:51.164976 1795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:51.165747 1795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [term 1 LEADER]: Becoming Leader. State: Replica: 7f48c246f143422282b2f669abfe33de, State: Running, Role: LEADER
I20260926 19:26:51.166221 1795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } }
I20260926 19:26:51.166903 1790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:26:51.168799 1797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f48c246f143422282b2f669abfe33de. Latest consensus state: current_term: 1 leader_uuid: "7f48c246f143422282b2f669abfe33de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } } }
I20260926 19:26:51.169129 1797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:51.169518 1796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f48c246f143422282b2f669abfe33de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f48c246f143422282b2f669abfe33de" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39175 } } }
I20260926 19:26:51.169922 1796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:51.171713 1801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:26:51.177559 1801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:26:51.186959 1801 catalog_manager.cc:1347] Generated new cluster ID: beba89b6a5964a62824845d731415ff1
I20260926 19:26:51.187101 1801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:26:51.198858 1801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:26:51.200183 1801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:26:51.207803 1801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f48c246f143422282b2f669abfe33de: Generated new TSK 0
I20260926 19:26:51.208896 1801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:26:51.258920 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:51.593382 1816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:51.593719 1816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:51.593891 1816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:51.600960 1816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:51.601106 1816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:51.601316 1816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:26:51.611645 1816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:51.612929 1816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:51.614342 1816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:51.627206 1822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:51.627156 1824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:51.627491 1821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:51.628083 1816 server_base.cc:1034] running on GCE node
I20260926 19:26:51.628692 1816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:51.629681 1816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:51.630903 1816 hybrid_clock.cc:648] HybridClock initialized: now 1790450811630858 us; error 55 us; skew 500 ppm
I20260926 19:26:51.631240 1816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:51 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450811, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:51.641676 1816 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:26:51.643018 1816 webserver.cc:460] Webserver started at http://127.1.39.1:44181/ using document root <none> and password file <none>
I20260926 19:26:51.643775 1816 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:51.643872 1816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:51.644122 1816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:51.646921 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "af5b13e70c5349aba21a052670a6b390"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "087ed083d3b4b0c9f1f200a248acc88e"
server_key_iv: "929cd0078684bef402a0e6baeefd3df2"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.647641 1816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "af5b13e70c5349aba21a052670a6b390"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "087ed083d3b4b0c9f1f200a248acc88e"
server_key_iv: "929cd0078684bef402a0e6baeefd3df2"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.653126 1816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:26:51.656662 1831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:51.658102 1816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:26:51.658308 1816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "af5b13e70c5349aba21a052670a6b390"
format_stamp: "Formatted at 2026-09-26 19:26:51 on dist-test-slave-9wb0"
server_key: "087ed083d3b4b0c9f1f200a248acc88e"
server_key_iv: "929cd0078684bef402a0e6baeefd3df2"
server_key_version: "encryptionkey@0"
I20260926 19:26:51.658517 1816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:51.679589 1816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:51.683519 1816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:51.683830 1816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:51.684791 1816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:51.686151 1816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:51.686259 1816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:51.686410 1816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:51.686480 1816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:51.720095 1816 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:39485
I20260926 19:26:51.720172 1943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:39485 every 8 connection(s)
I20260926 19:26:51.721458 1816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:26:51.730670 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1816
I20260926 19:26:51.730896 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:26:51.731653 1180 external_mini_cluster.cc:1428] Setting key 2254faa9f99e9ae3dbd82a886286e2a4
Sep 26 19:26:51 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450811, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:51.742865 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:51.724067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:48859 (local address 127.1.39.62:39175)
0926 19:26:51.724488 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:51.724493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:51.725518 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 19:26:51.727398 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:51.727404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:51.727509 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:51.727634 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:51.729098 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.729888 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.732236 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.732791 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.732899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:51.736614 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:51.736623 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:51.736631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:51.736676 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:51.739490 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:51.740437 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:51.740441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:51.740444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:51.740572 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:51.741055 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:51.741058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:51.741060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:51.741430 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:51.741559 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:51.742457 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 19:26:51.742615 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":212,"threads_started":1}
I20260926 19:26:51.744498 1947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:51.724632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39175 (local address 127.1.39.1:48859)
0926 19:26:51.725363 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:51.725404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:51.727134 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:51.727873 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:51.727881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:51.728371 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:51.728923 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:51.728940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.730045 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.730052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:51.732096 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:51.732110 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:51.733016 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:51.733023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:51.733321 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:51.734045 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:51.734073 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:51.736385 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:51.739636 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:51.739648 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:51.739668 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:51.740291 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:51.740694 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:51.740702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:51.740708 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:51.740947 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:51.741558 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:51.741568 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:51.742032 (+ 464us) client_negotiation.cc:769] Sending connection context
0926 19:26:51.742363 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 19:26:51.742956 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":260,"threads_started":1}
I20260926 19:26:51.746310 1944 heartbeater.cc:347] Connected to a master server at 127.1.39.62:39175
I20260926 19:26:51.746621 1944 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:51.747327 1944 heartbeater.cc:511] Master 127.1.39.62:39175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:26:51.749507 1755 ts_manager.cc:194] Registered new tserver with Master: af5b13e70c5349aba21a052670a6b390 (127.1.39.1:39485)
I20260926 19:26:51.750933 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:48859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:26:51.773983 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:52.092285 1950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:52.092624 1950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:52.092803 1950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:52.101532 1950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:52.101680 1950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:52.101873 1950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:26:52.111903 1950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:52.113233 1950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:52.114598 1950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:52.127614 1955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:52.127468 1956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:52.127878 1958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:52.128713 1950 server_base.cc:1034] running on GCE node
I20260926 19:26:52.129441 1950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:52.130395 1950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:52.131614 1950 hybrid_clock.cc:648] HybridClock initialized: now 1790450812131565 us; error 45 us; skew 500 ppm
I20260926 19:26:52.131914 1950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:52 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:52.134990 1950 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:26:52.138651 1950 webserver.cc:460] Webserver started at http://127.1.39.2:36603/ using document root <none> and password file <none>
I20260926 19:26:52.139340 1950 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:52.139492 1950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:52.139806 1950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:52.142786 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "725b030d10ce4099b1dd781de09a8bd0"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "ab65c013624726a81117fa4106f1735d"
server_key_iv: "644d26c3d19c4abe26c0454986852c68"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.143571 1950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "725b030d10ce4099b1dd781de09a8bd0"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "ab65c013624726a81117fa4106f1735d"
server_key_iv: "644d26c3d19c4abe26c0454986852c68"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.148643 1950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:26:52.152118 1965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.153476 1950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 19:26:52.153647 1950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "725b030d10ce4099b1dd781de09a8bd0"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "ab65c013624726a81117fa4106f1735d"
server_key_iv: "644d26c3d19c4abe26c0454986852c68"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.153834 1950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:52.177250 1950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:52.181267 1950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:52.181578 1950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:52.182586 1950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:52.183883 1950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:52.183990 1950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.184141 1950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:52.184213 1950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.217315 1950 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:36329
I20260926 19:26:52.217463 2077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:36329 every 8 connection(s)
I20260926 19:26:52.218654 1950 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:26:52.220834 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 1950
I20260926 19:26:52.221069 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:26:52.221577 1180 external_mini_cluster.cc:1428] Setting key 814fea39486d0c823b3dd06b2cdb5977
Sep 26 19:26:52 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:26:52.240690 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.220995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:36837 (local address 127.1.39.62:39175)
0926 19:26:52.221211 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.221218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.222605 (+ 1387us) server_negotiation.cc:407] Connection header received
0926 19:26:52.225060 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.225064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.225128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.225216 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:52.227456 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.228149 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.229867 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.230703 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.230845 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.234556 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:52.234565 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:52.234574 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:52.234619 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:52.237505 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.238410 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.238414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.238416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.238568 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.239048 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.239051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.239053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.239345 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:52.239460 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.240234 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.240432 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
I20260926 19:26:52.241886 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.221559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39175 (local address 127.1.39.2:36837)
0926 19:26:52.222425 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:52.222477 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:52.224727 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:52.225527 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:52.225541 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:52.226379 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:52.227259 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.227297 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.228315 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.228323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.229735 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.229746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.230939 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.230946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.231147 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.231844 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:52.231875 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:52.234297 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:52.237642 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:52.237655 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:52.237676 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:52.238287 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:52.238684 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:52.238692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:52.238699 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:52.238937 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:52.239482 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:52.239493 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:52.239849 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 19:26:52.240078 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 19:26:52.240544 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":302,"threads_started":1}
I20260926 19:26:52.243305 2078 heartbeater.cc:347] Connected to a master server at 127.1.39.62:39175
I20260926 19:26:52.243639 2078 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:52.244549 2078 heartbeater.cc:511] Master 127.1.39.62:39175 requested a full tablet report, sending...
I20260926 19:26:52.246049 1755 ts_manager.cc:194] Registered new tserver with Master: 725b030d10ce4099b1dd781de09a8bd0 (127.1.39.2:36329)
I20260926 19:26:52.247258 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:36837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:26:52.265376 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:52.589026 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:52.589282 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:52.589483 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:52.596562 2083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:52.596664 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:52.596827 2083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:26:52.606557 2083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:39175
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:52.607823 2083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:52.609133 2083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:52.624440 2088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:52.624486 2091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:52.624533 2089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:52.625790 2083 server_base.cc:1034] running on GCE node
I20260926 19:26:52.626396 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:52.627367 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:52.628576 2083 hybrid_clock.cc:648] HybridClock initialized: now 1790450812628485 us; error 84 us; skew 500 ppm
I20260926 19:26:52.628862 2083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:52 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:52.631927 2083 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:26:52.633495 2083 webserver.cc:460] Webserver started at http://127.1.39.3:44243/ using document root <none> and password file <none>
I20260926 19:26:52.634147 2083 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:52.634243 2083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:52.634501 2083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:52.637382 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "613b90f153dc40788e405aacdb4b66d2"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "295ecf5d52fb61b431fe2a2f76ed0c58"
server_key_iv: "436d29a087d948a1b59b96810a4aa2ad"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.638118 2083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "613b90f153dc40788e405aacdb4b66d2"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "295ecf5d52fb61b431fe2a2f76ed0c58"
server_key_iv: "436d29a087d948a1b59b96810a4aa2ad"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.643177 2083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:26:52.646669 2098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.648015 2083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:26:52.648147 2083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "613b90f153dc40788e405aacdb4b66d2"
format_stamp: "Formatted at 2026-09-26 19:26:52 on dist-test-slave-9wb0"
server_key: "295ecf5d52fb61b431fe2a2f76ed0c58"
server_key_iv: "436d29a087d948a1b59b96810a4aa2ad"
server_key_version: "encryptionkey@0"
I20260926 19:26:52.648305 2083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:52.665019 2083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:52.669276 2083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:52.669595 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:52.670687 2083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:26:52.672338 2083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:52.672410 2083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.672564 2083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:52.672617 2083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:52.705371 2083 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:40055
I20260926 19:26:52.705482 2210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:40055 every 8 connection(s)
I20260926 19:26:52.706712 2083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:26:52.712620 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2083
I20260926 19:26:52.712818 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:26:52.713387 1180 external_mini_cluster.cc:1428] Setting key 0374e57778d14b9e1bd400055cc72672
Sep 26 19:26:52 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450812, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:52.730798 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.709486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:54447 (local address 127.1.39.62:39175)
0926 19:26:52.709665 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.709670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.710781 (+ 1111us) server_negotiation.cc:407] Connection header received
0926 19:26:52.713494 (+ 2713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.713501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.713593 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.713686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:52.715786 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.716890 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.718500 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.718863 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.718958 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.722771 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:52.722777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:52.722783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:52.722822 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:52.726259 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.727012 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.727018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.727021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.727188 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.727875 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.727885 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.727891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.728365 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:52.728592 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.729825 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.730473 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:26:52.731761 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.709915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39175 (local address 127.1.39.3:54447)
0926 19:26:52.710615 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:52.710660 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:52.713080 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:52.713993 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:52.714003 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:52.714717 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:52.715581 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.715611 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.717032 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.717036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.718372 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.718384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.719041 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.719046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.719252 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.720043 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:52.720067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:52.722524 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:52.726391 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:52.726397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:52.726410 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:52.726816 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:52.727329 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:52.727335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:52.727339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:52.727687 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:52.728561 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:52.728569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:52.729002 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 19:26:52.729310 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 19:26:52.729916 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":265,"threads_started":1}
I20260926 19:26:52.733780 2211 heartbeater.cc:347] Connected to a master server at 127.1.39.62:39175
I20260926 19:26:52.734138 2211 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:52.735102 2211 heartbeater.cc:511] Master 127.1.39.62:39175 requested a full tablet report, sending...
I20260926 19:26:52.737000 1755 ts_manager.cc:194] Registered new tserver with Master: 613b90f153dc40788e405aacdb4b66d2 (127.1.39.3:40055)
I20260926 19:26:52.738238 1755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:54447
I20260926 19:26:52.742877 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:26:52.754735 1944 heartbeater.cc:503] Master 127.1.39.62:39175 was elected leader, sending a full tablet report...
I20260926 19:26:52.765931 1946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.749144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.1.39.62:39175)
0926 19:26:52.749396 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.749405 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.750060 (+ 655us) server_negotiation.cc:407] Connection header received
0926 19:26:52.750270 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.750273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.750333 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.750412 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:52.752728 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.753601 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.755889 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.756401 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.756534 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.760108 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:52.760119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:52.760122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:52.760151 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:52.763017 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.764049 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.764053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.764055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.764176 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:52.764613 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:52.764618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:52.764620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:52.764900 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:52.765072 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.765555 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.765712 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:26:52.772914 1755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:26:52.780089 1755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:26:52.821581 2225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.794728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:39485 (local address 127.0.0.1:50900)
0926 19:26:52.801854 (+ 7126us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:52.801899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:52.802061 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:52.812846 (+ 10785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:52.812852 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:52.812877 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:52.813272 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.813286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.815900 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.815906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.817986 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.817999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.820360 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.820367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.820576 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.820632 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 19:26:52.820745 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 19:26:52.821061 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6881,"mutex_wait_us":21,"thread_start_us":6734,"threads_started":1}
I20260926 19:26:52.821800 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.793750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.1.39.2:36329)
0926 19:26:52.806198 (+ 12448us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.806208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.810091 (+ 3883us) server_negotiation.cc:407] Connection header received
0926 19:26:52.810230 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.810239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.810556 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.810765 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:52.812911 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.814632 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.818436 (+ 3804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.819958 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.820085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.820315 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.820990 (+ 675us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.821156 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12258,"thread_start_us":12447,"threads_started":1}
I20260926 19:26:52.822242 2226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.795399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:36329 (local address 127.0.0.1:47174)
0926 19:26:52.809796 (+ 14397us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:52.809818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:52.809945 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:52.812277 (+ 2332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:52.812285 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:52.812319 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:52.812708 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.812720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.814792 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.814799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.818269 (+ 3470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.818282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.820444 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.820450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.820663 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.820686 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:26:52.820761 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 19:26:52.821455 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14261,"mutex_wait_us":413,"thread_start_us":14119,"threads_started":1}
I20260926 19:26:52.822676 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.793837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.1.39.1:39485)
0926 19:26:52.811590 (+ 17753us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.811603 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.811636 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:26:52.811766 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.811773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.812007 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.813560 (+ 1553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:52.813781 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.815734 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.818653 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.820141 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.821813 (+ 1672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.822081 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.822178 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.822355 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17549,"thread_start_us":17393,"threads_started":1}
I20260926 19:26:52.824810 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.810293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42850 (local address 127.1.39.3:40055)
0926 19:26:52.811951 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:52.811961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:52.812721 (+ 760us) server_negotiation.cc:407] Connection header received
0926 19:26:52.812975 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:52.812986 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:52.813252 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:52.813461 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:52.814559 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.817015 (+ 2456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.820535 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.822461 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.822591 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.822852 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:52.824281 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0926 19:26:52.824479 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":1310,"threads_started":1}
I20260926 19:26:52.825613 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:52.811913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40055 (local address 127.0.0.1:42850)
0926 19:26:52.812577 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:52.812600 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:52.812757 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:52.813799 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:52.813806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:52.813836 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:52.814331 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.814346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.817724 (+ 3378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.817732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.820320 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:52.820341 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:52.823907 (+ 3566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:52.823914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:52.824081 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:52.824112 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:26:52.825047 (+ 935us) client_negotiation.cc:240] Negotiation successful
0926 19:26:52.825184 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1,"thread_start_us":314,"threads_started":1}
I20260926 19:26:52.827584 2013 tablet_service.cc:1467] Processing CreateTablet for tablet b4b4242774b640f69b4289047d170991 (DEFAULT_TABLE table=test-table [id=e07f26e3530d4b728e49241a0c82c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:52.828294 1879 tablet_service.cc:1467] Processing CreateTablet for tablet b4b4242774b640f69b4289047d170991 (DEFAULT_TABLE table=test-table [id=e07f26e3530d4b728e49241a0c82c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:52.829180 2013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b4242774b640f69b4289047d170991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:52.828954 2146 tablet_service.cc:1467] Processing CreateTablet for tablet b4b4242774b640f69b4289047d170991 (DEFAULT_TABLE table=test-table [id=e07f26e3530d4b728e49241a0c82c27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:52.830137 1879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b4242774b640f69b4289047d170991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:52.830505 2146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4b4242774b640f69b4289047d170991. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:52.851507 2229 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Bootstrap starting.
I20260926 19:26:52.855019 2230 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Bootstrap starting.
I20260926 19:26:52.858551 2230 tablet_bootstrap.cc:654] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:52.862254 2229 tablet_bootstrap.cc:654] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:52.862660 2230 log.cc:824] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:52.863554 2231 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Bootstrap starting.
I20260926 19:26:52.863772 2229 log.cc:824] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:52.867274 2231 tablet_bootstrap.cc:654] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:52.867472 2229 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: No bootstrap required, opened a new log
I20260926 19:26:52.867792 2229 ts_tablet_manager.cc:1397] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20260926 19:26:52.868433 2231 log.cc:824] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:52.879034 2231 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: No bootstrap required, opened a new log
I20260926 19:26:52.879402 2231 ts_tablet_manager.cc:1397] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20260926 19:26:52.881320 2229 raft_consensus.cc:348] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.882773 2229 raft_consensus.cc:374] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:52.882992 2229 raft_consensus.cc:729] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af5b13e70c5349aba21a052670a6b390, State: Initialized, Role: FOLLOWER
I20260926 19:26:52.883841 2229 consensus_queue.cc:260] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.884661 2230 tablet_bootstrap.cc:492] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: No bootstrap required, opened a new log
I20260926 19:26:52.885046 2230 ts_tablet_manager.cc:1397] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.005s
I20260926 19:26:52.892149 2231 raft_consensus.cc:348] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.892820 2231 raft_consensus.cc:374] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:52.892992 2231 raft_consensus.cc:729] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725b030d10ce4099b1dd781de09a8bd0, State: Initialized, Role: FOLLOWER
I20260926 19:26:52.895804 2231 consensus_queue.cc:260] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.897714 2229 ts_tablet_manager.cc:1428] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Time spent starting tablet: real 0.030s user 0.018s sys 0.010s
I20260926 19:26:52.899031 2230 raft_consensus.cc:348] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.899660 2230 raft_consensus.cc:374] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:52.899827 2230 raft_consensus.cc:729] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613b90f153dc40788e405aacdb4b66d2, State: Initialized, Role: FOLLOWER
I20260926 19:26:52.900761 2230 consensus_queue.cc:260] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:52.904866 2078 heartbeater.cc:503] Master 127.1.39.62:39175 was elected leader, sending a full tablet report...
I20260926 19:26:52.905050 2231 ts_tablet_manager.cc:1428] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Time spent starting tablet: real 0.025s user 0.015s sys 0.005s
I20260926 19:26:52.905450 2230 ts_tablet_manager.cc:1428] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260926 19:26:52.905671 2211 heartbeater.cc:503] Master 127.1.39.62:39175 was elected leader, sending a full tablet report...
W20260926 19:26:52.958510 2212 tablet.cc:2369] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:26:52.970638 2079 tablet.cc:2369] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:26:52.973711 1945 tablet.cc:2369] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:26:53.081590 2235 raft_consensus.cc:482] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:26:53.082011 2235 raft_consensus.cc:504] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:53.083726 2235 leader_election.cc:288] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 725b030d10ce4099b1dd781de09a8bd0 (127.1.39.2:36329), 613b90f153dc40788e405aacdb4b66d2 (127.1.39.3:40055)
I20260926 19:26:53.091065 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.084134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:52477 (local address 127.1.39.2:36329)
0926 19:26:53.084379 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.084388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.084922 (+ 534us) server_negotiation.cc:407] Connection header received
0926 19:26:53.085231 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.085240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.085360 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.085458 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:53.086018 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.086740 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.089049 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.090090 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.090253 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.090394 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.090637 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.090759 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:26:53.091109 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.084151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:36329 (local address 127.1.39.1:52477)
0926 19:26:53.084796 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:53.084813 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:53.085029 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:53.085506 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:53.085512 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:53.085536 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:53.085873 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:53.085882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.086879 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.086886 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:53.088893 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:53.088904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.090301 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.090307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:53.090476 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.090500 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:26:53.090612 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 19:26:53.090754 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":357,"threads_started":1}
I20260926 19:26:53.091565 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.085208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:48083 (local address 127.1.39.3:40055)
0926 19:26:53.085472 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.085478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.085777 (+ 299us) server_negotiation.cc:407] Connection header received
0926 19:26:53.085975 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.085979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.086034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.086131 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:53.086615 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.087306 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.089791 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.090556 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.090706 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.090809 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.091192 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.091317 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 19:26:53.091671 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.084990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40055 (local address 127.1.39.1:48083)
0926 19:26:53.085678 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:53.085694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:53.085842 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:53.086144 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:53.086149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:53.086170 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:53.086463 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:53.086471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.087443 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.087450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:53.089624 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:53.089639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.090769 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.090775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:53.090929 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.090952 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:26:53.091171 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 19:26:53.091310 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":308,"threads_started":1}
I20260926 19:26:53.092414 2033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b4242774b640f69b4289047d170991" candidate_uuid: "af5b13e70c5349aba21a052670a6b390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725b030d10ce4099b1dd781de09a8bd0" is_pre_election: true
I20260926 19:26:53.092824 2166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b4242774b640f69b4289047d170991" candidate_uuid: "af5b13e70c5349aba21a052670a6b390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613b90f153dc40788e405aacdb4b66d2" is_pre_election: true
I20260926 19:26:53.093088 2033 raft_consensus.cc:2454] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af5b13e70c5349aba21a052670a6b390 in term 0.
I20260926 19:26:53.093403 2166 raft_consensus.cc:2454] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af5b13e70c5349aba21a052670a6b390 in term 0.
I20260926 19:26:53.093888 1834 leader_election.cc:302] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 725b030d10ce4099b1dd781de09a8bd0, af5b13e70c5349aba21a052670a6b390; no voters:
I20260926 19:26:53.094286 2235 raft_consensus.cc:2781] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:26:53.094432 2235 raft_consensus.cc:482] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:26:53.094561 2235 raft_consensus.cc:3037] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:53.096758 2235 raft_consensus.cc:504] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:53.097368 2235 leader_election.cc:288] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [CANDIDATE]: Term 1 election: Requested vote from peers 725b030d10ce4099b1dd781de09a8bd0 (127.1.39.2:36329), 613b90f153dc40788e405aacdb4b66d2 (127.1.39.3:40055)
I20260926 19:26:53.097903 2166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b4242774b640f69b4289047d170991" candidate_uuid: "af5b13e70c5349aba21a052670a6b390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613b90f153dc40788e405aacdb4b66d2"
I20260926 19:26:53.097883 2033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4b4242774b640f69b4289047d170991" candidate_uuid: "af5b13e70c5349aba21a052670a6b390" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725b030d10ce4099b1dd781de09a8bd0"
I20260926 19:26:53.098101 2166 raft_consensus.cc:3037] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:53.098097 2033 raft_consensus.cc:3037] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:53.101766 2166 raft_consensus.cc:2454] T b4b4242774b640f69b4289047d170991 P 613b90f153dc40788e405aacdb4b66d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af5b13e70c5349aba21a052670a6b390 in term 1.
I20260926 19:26:53.101766 2033 raft_consensus.cc:2454] T b4b4242774b640f69b4289047d170991 P 725b030d10ce4099b1dd781de09a8bd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af5b13e70c5349aba21a052670a6b390 in term 1.
I20260926 19:26:53.102157 1834 leader_election.cc:302] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 725b030d10ce4099b1dd781de09a8bd0, af5b13e70c5349aba21a052670a6b390; no voters:
I20260926 19:26:53.102491 2235 raft_consensus.cc:2781] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:53.103008 2235 raft_consensus.cc:686] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [term 1 LEADER]: Becoming Leader. State: Replica: af5b13e70c5349aba21a052670a6b390, State: Running, Role: LEADER
I20260926 19:26:53.103641 2235 consensus_queue.cc:237] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } }
I20260926 19:26:53.108120 1755 catalog_manager.cc:5495] T b4b4242774b640f69b4289047d170991 P af5b13e70c5349aba21a052670a6b390 reported cstate change: term changed from 0 to 1, leader changed from <none> to af5b13e70c5349aba21a052670a6b390 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "af5b13e70c5349aba21a052670a6b390" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af5b13e70c5349aba21a052670a6b390" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 39485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "725b030d10ce4099b1dd781de09a8bd0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613b90f153dc40788e405aacdb4b66d2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40055 } health_report { overall_health: UNKNOWN } } }
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:53.180104 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.163264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.1.39.1:39485)
0926 19:26:53.163478 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.163483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.164433 (+ 950us) server_negotiation.cc:407] Connection header received
0926 19:26:53.164676 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.164680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.164747 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.164861 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.166512 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.167373 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.168846 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.169274 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.169481 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.173772 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.173786 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.173803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.173848 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.177205 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.177906 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.177925 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.177928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.178098 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.178504 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.178514 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.178518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.178851 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.179166 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.179502 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.179772 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:53.226423 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.212013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.1.39.1:39485)
0926 19:26:53.212249 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.212259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.212581 (+ 322us) server_negotiation.cc:407] Connection header received
0926 19:26:53.212805 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.212809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.212869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.212972 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.214395 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.215182 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.216388 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.216842 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.216988 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.220900 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.220905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.220908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.220939 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.223547 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.224173 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.224181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.224184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.224396 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.224972 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.224977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.224980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.225245 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.225595 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.225852 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.226084 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:53.272471 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.258288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.1.39.1:39485)
0926 19:26:53.258480 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.258489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.258811 (+ 322us) server_negotiation.cc:407] Connection header received
0926 19:26:53.259018 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.259028 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.259111 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.259220 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.260522 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.261465 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.262765 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.263206 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.263348 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.267424 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.267431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.267435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.267474 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.270173 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.270853 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.270857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.270860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.270986 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.271316 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.271319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.271321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.271621 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.271807 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.272052 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.272217 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 19:26:53.274610 1859 tablet_service.cc:2908] Rejecting scan request for tablet b4b4242774b640f69b4289047d170991: Uninitialized: clean time has not yet been initialized
I20260926 19:26:53.277438 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1816
I20260926 19:26:53.289891 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.278359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50956 (local address 127.1.39.1:39485)
0926 19:26:53.278568 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.278577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.278826 (+ 249us) server_negotiation.cc:407] Connection header received
0926 19:26:53.278940 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.278943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.278999 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.279079 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.280214 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.280915 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.282135 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.282529 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.282622 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.284795 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.284799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.284802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.284830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.287704 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.288298 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.288302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.288304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.288408 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.288701 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.288704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.288706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.288931 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.289097 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.289394 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.289568 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:26:53.290609 1939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:53.382630 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1816
I20260926 19:26:53.400384 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1950
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:26:53.414433 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.401121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.1.39.2:36329)
0926 19:26:53.401282 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.401287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.401346 (+ 59us) server_negotiation.cc:407] Connection header received
0926 19:26:53.401440 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.401444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.401521 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.401605 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.402729 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.403433 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.404703 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.405097 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.405193 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.409343 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.409365 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.409380 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.409414 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.411925 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.412569 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.412576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.412578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.412703 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.413069 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.413072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.413074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.413487 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.413707 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.414001 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.414172 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:26:53.415113 2073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:53.507957 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1950
I20260926 19:26:53.524173 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2083
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[1710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:26:53.538568 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:53.524929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.1.39.3:40055)
0926 19:26:53.525119 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:53.525124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:53.525143 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:26:53.525257 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:53.525260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:53.525460 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:53.525553 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:53.526739 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.527467 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.528643 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:53.529009 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:53.529101 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:53.533171 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:53.533186 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:53.533202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:53.533235 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:53.535810 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.536419 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.536426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.536428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.536588 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:53.537053 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:53.537057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:53.537060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:53.537511 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:53.537748 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:53.538018 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 19:26:53.538197 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 19:26:53.539230 2206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:53.627861 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2083
I20260926 19:26:53.644315 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 1723
I20260926 19:26:53.645516 1785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:53.754037 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 1723
2026-09-26T19:26:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4325 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[2259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[2259](info): set up 2 sockets
Sep 26 19:26:53 dist-test-slave-9wb0 krb5kdc[2259](info): commencing operation
krb5kdc: starting...
W20260926 19:26:54.853540 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:26:54 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:26:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:26:55.041765 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40879
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:40879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:55.365967 2272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:55.366257 2272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:55.366349 2272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:55.373379 2272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:26:55.373486 2272 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:26:55.373530 2272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:55.373601 2272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:55.373651 2272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:26:55.373703 2272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:26:55.383230 2272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:40879
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:55.384426 2272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:55.385761 2272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:55.397661 2280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:55.397648 2277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:55.397648 2278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:55.398316 2272 server_base.cc:1034] running on GCE node
I20260926 19:26:55.399349 2272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:55.400846 2272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:55.402088 2272 hybrid_clock.cc:648] HybridClock initialized: now 1790450815402018 us; error 46 us; skew 500 ppm
I20260926 19:26:55.402429 2272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:55 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:55.405587 2272 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:26:55.407038 2272 webserver.cc:460] Webserver started at http://127.1.39.62:36869/ using document root <none> and password file <none>
I20260926 19:26:55.407804 2272 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:55.407938 2272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:55.408262 2272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:55.411229 2272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "40b1197f9f8a4ce381757dfbb12b8e9a"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "cfb372429a1ea9e57585945ea65d497b"
server_key_iv: "02400f943bc170413b689fb9b9d75379"
server_key_version: "encryptionkey@0"
I20260926 19:26:55.412031 2272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "40b1197f9f8a4ce381757dfbb12b8e9a"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "cfb372429a1ea9e57585945ea65d497b"
server_key_iv: "02400f943bc170413b689fb9b9d75379"
server_key_version: "encryptionkey@0"
I20260926 19:26:55.417068 2272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:26:55.420693 2287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:55.422096 2272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:26:55.422295 2272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "40b1197f9f8a4ce381757dfbb12b8e9a"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "cfb372429a1ea9e57585945ea65d497b"
server_key_iv: "02400f943bc170413b689fb9b9d75379"
server_key_version: "encryptionkey@0"
I20260926 19:26:55.422739 2272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:55.445140 2272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:55.449910 2272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:55.450214 2272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:55.472190 2272 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:40879
I20260926 19:26:55.472211 2348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:40879 every 8 connection(s)
I20260926 19:26:55.473673 2272 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:26:55.477458 2349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:55.479246 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2272
I20260926 19:26:55.479482 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:26:55.480084 1180 external_mini_cluster.cc:1428] Setting key e5995868b03483cf5fafbe748c776351
I20260926 19:26:55.490522 2349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Bootstrap starting.
I20260926 19:26:55.494186 2349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:55.495302 2349 log.cc:824] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:26:55 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:55.498314 2349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: No bootstrap required, opened a new log
I20260926 19:26:55.503757 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:55.482204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.1.39.62:40879)
0926 19:26:55.483041 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:55.483056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:55.483116 (+ 60us) server_negotiation.cc:407] Connection header received
0926 19:26:55.485502 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:55.485529 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:55.486036 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:55.486503 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:55.487835 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:55.489852 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:55.491208 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:55.492257 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:55.492444 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:55.496687 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:55.496695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:55.496708 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:55.496740 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:55.499734 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:55.500521 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:55.500527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:55.500533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:55.500695 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:55.501087 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:55.501090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:55.501092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:55.501661 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:55.501889 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:55.502346 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 19:26:55.502711 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260926 19:26:55.507448 2349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } }
I20260926 19:26:55.507797 2349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:55.507893 2349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b1197f9f8a4ce381757dfbb12b8e9a, State: Initialized, Role: FOLLOWER
I20260926 19:26:55.508488 2349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } }
I20260926 19:26:55.508723 2349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:26:55.508849 2349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:26:55.509012 2349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:55.511406 2349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } }
I20260926 19:26:55.511912 2349 leader_election.cc:302] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40b1197f9f8a4ce381757dfbb12b8e9a; no voters:
I20260926 19:26:55.512475 2349 leader_election.cc:288] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:26:55.512609 2354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:55.513334 2354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [term 1 LEADER]: Becoming Leader. State: Replica: 40b1197f9f8a4ce381757dfbb12b8e9a, State: Running, Role: LEADER
I20260926 19:26:55.513929 2354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } }
I20260926 19:26:55.514678 2349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:26:55.516837 2355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } } }
I20260926 19:26:55.517273 2355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:55.517215 2356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40b1197f9f8a4ce381757dfbb12b8e9a. Latest consensus state: current_term: 1 leader_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40b1197f9f8a4ce381757dfbb12b8e9a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40879 } } }
I20260926 19:26:55.517539 2356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:55.519039 2363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:26:55.524151 2363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:26:55.533396 2363 catalog_manager.cc:1347] Generated new cluster ID: 82f4fc69496e437dae053a6fd46b3c60
I20260926 19:26:55.533519 2363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:26:55.548538 2363 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:26:55.550268 2363 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:26:55.561771 2363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Generated new TSK 0
I20260926 19:26:55.562711 2363 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:26:55.617097 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40879
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:55.954687 2375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:55.954970 2375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:55.955142 2375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:55.962308 2375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:55.962421 2375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:55.962568 2375 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:26:55.962656 2375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:26:55.972374 2375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:55.973745 2375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:55.975121 2375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:55.988735 2380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:55.988667 2381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:55.988708 2383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:55.989511 2375 server_base.cc:1034] running on GCE node
I20260926 19:26:55.990170 2375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:55.991078 2375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:55.992323 2375 hybrid_clock.cc:648] HybridClock initialized: now 1790450815992256 us; error 45 us; skew 500 ppm
I20260926 19:26:55.992702 2375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:55 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:55.995889 2375 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:26:55.997597 2375 webserver.cc:460] Webserver started at http://127.1.39.1:36403/ using document root <none> and password file <none>
I20260926 19:26:55.998315 2375 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:55.998448 2375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:55.998778 2375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:56.001749 2375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "27249121908d4ffb86138033c31bbaf2"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "3672d483dc0f66cd417c51920377a74a"
server_key_iv: "3a8a12780f62e71c2765f4c1adc3bd92"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.002614 2375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "27249121908d4ffb86138033c31bbaf2"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "3672d483dc0f66cd417c51920377a74a"
server_key_iv: "3a8a12780f62e71c2765f4c1adc3bd92"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.008002 2375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:26:56.011607 2390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:56.013156 2375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:26:56.013394 2375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "27249121908d4ffb86138033c31bbaf2"
format_stamp: "Formatted at 2026-09-26 19:26:55 on dist-test-slave-9wb0"
server_key: "3672d483dc0f66cd417c51920377a74a"
server_key_iv: "3a8a12780f62e71c2765f4c1adc3bd92"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.013630 2375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:56.039604 2375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:56.043752 2375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:56.044046 2375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:56.047014 2375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:56.047132 2375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:56.047269 2375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:56.047374 2375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:26:56 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:56.072151 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.047703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:35123 (local address 127.1.39.62:40879)
0926 19:26:56.048198 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:56.048206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:56.051044 (+ 2838us) server_negotiation.cc:407] Connection header received
0926 19:26:56.053219 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:56.053224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:56.053371 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:56.053491 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:56.056106 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.057446 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.060039 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.060498 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.060657 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.065588 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:56.065593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:56.065596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:56.065629 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:56.068755 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.069543 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.069550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.069553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.069725 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.070342 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.070349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.070355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.070713 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:56.070865 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:56.071563 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 19:26:56.071771 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":258,"threads_started":1}
I20260926 19:26:56.073201 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.049793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.1:35123)
0926 19:26:56.050757 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:56.050799 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:56.052842 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:56.053982 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:56.053996 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:56.054922 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:56.055862 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.055895 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.057626 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.057633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.059868 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.059883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.060992 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.061006 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.061383 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.062311 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:56.062352 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:56.065332 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:56.068897 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.068906 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.068924 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.069417 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.069859 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.069870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.069875 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.070194 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.070847 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:56.070854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:56.071214 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 19:26:56.071476 (+ 262us) client_negotiation.cc:240] Negotiation successful
0926 19:26:56.072008 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":461,"threads_started":1}
I20260926 19:26:56.099783 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.083941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:32907 (local address 127.1.39.62:40879)
0926 19:26:56.084106 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:56.084110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:56.084736 (+ 626us) server_negotiation.cc:407] Connection header received
0926 19:26:56.084959 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:56.084965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:56.085078 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:56.085191 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:56.086698 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.087374 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.089901 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.090459 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.090603 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.093353 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:56.093362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:56.093369 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:56.093413 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:56.095988 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.097015 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.097022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.097025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.097193 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.097833 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.097837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.097839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.098237 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:56.098459 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:56.099143 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 19:26:56.099371 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:26:56.099978 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.084014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.1:32907)
0926 19:26:56.084625 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:56.084640 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:56.084804 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:56.085250 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:56.085259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:56.086062 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:56.086531 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.086547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.087509 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.087517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.089660 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.089686 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.090692 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.090702 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.090980 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.091821 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:56.091839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:56.093070 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:56.096125 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.096133 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.096139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.096854 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.097381 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.097387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.097392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.097639 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.098542 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:56.098545 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:56.098837 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 19:26:56.099027 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 19:26:56.099417 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":360,"threads_started":1}
I20260926 19:26:56.109107 2375 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:34543
I20260926 19:26:56.109123 2510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:34543 every 8 connection(s)
I20260926 19:26:56.110338 2375 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:26:56.112564 2511 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40879
I20260926 19:26:56.112807 2511 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:56.113626 2511 heartbeater.cc:511] Master 127.1.39.62:40879 requested a full tablet report, sending...
I20260926 19:26:56.116214 2304 ts_manager.cc:194] Registered new tserver with Master: 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543)
I20260926 19:26:56.117805 2304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:35123
I20260926 19:26:56.118464 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2375
I20260926 19:26:56.118736 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:26:56.119494 1180 external_mini_cluster.cc:1428] Setting key 1c58fea9f6254ce76b567bb8295d8d60
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:26:56.158758 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40879
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:56.497713 2515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:56.498010 2515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:56.498183 2515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:56.505623 2515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:56.505760 2515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:56.505903 2515 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:26:56.506034 2515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:26:56.515894 2515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:56.517208 2515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:56.518672 2515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:56.531602 2520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:56.531690 2521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:56.531723 2523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:56.532258 2515 server_base.cc:1034] running on GCE node
I20260926 19:26:56.532910 2515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:56.533923 2515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:56.535215 2515 hybrid_clock.cc:648] HybridClock initialized: now 1790450816535014 us; error 172 us; skew 500 ppm
I20260926 19:26:56.535631 2515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:56 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450816, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:56.538913 2515 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:26:56.540385 2515 webserver.cc:460] Webserver started at http://127.1.39.2:39273/ using document root <none> and password file <none>
I20260926 19:26:56.541078 2515 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:56.541200 2515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:56.541559 2515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:56.544353 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "0d4c66e24123465d969238c328ab2496"
format_stamp: "Formatted at 2026-09-26 19:26:56 on dist-test-slave-9wb0"
server_key: "bca686e36ca08e30d9110c3c94510f11"
server_key_iv: "505724e6891b6cc58b1736d025cd8066"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.545188 2515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "0d4c66e24123465d969238c328ab2496"
format_stamp: "Formatted at 2026-09-26 19:26:56 on dist-test-slave-9wb0"
server_key: "bca686e36ca08e30d9110c3c94510f11"
server_key_iv: "505724e6891b6cc58b1736d025cd8066"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.550153 2515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:26:56.553526 2530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:56.554843 2515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:26:56.555016 2515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "0d4c66e24123465d969238c328ab2496"
format_stamp: "Formatted at 2026-09-26 19:26:56 on dist-test-slave-9wb0"
server_key: "bca686e36ca08e30d9110c3c94510f11"
server_key_iv: "505724e6891b6cc58b1736d025cd8066"
server_key_version: "encryptionkey@0"
I20260926 19:26:56.555261 2515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:56.576602 2515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:56.580876 2515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:56.581157 2515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:56.583734 2515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:56.583841 2515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:56.583971 2515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:56.584053 2515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:26:56 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450816, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:56.607879 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.584492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:54127 (local address 127.1.39.62:40879)
0926 19:26:56.584678 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:56.584686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:56.586215 (+ 1529us) server_negotiation.cc:407] Connection header received
0926 19:26:56.588154 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:56.588160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:56.588246 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:56.588370 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:56.590668 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.591957 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.595172 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.595768 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.595904 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.601409 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:56.601414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:56.601417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:56.601449 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:56.603944 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.604999 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.605007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.605014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.605136 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.605706 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.605712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.605717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.606135 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:56.606290 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:56.607456 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0926 19:26:56.607642 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:26:56.610009 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.585135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.2:54127)
0926 19:26:56.586056 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:56.586095 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:56.587869 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:56.588668 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:56.588681 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:56.589559 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:56.590385 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.590403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.592177 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.592182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.595002 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.595021 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.596281 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.596292 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.596608 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.597472 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:56.597543 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:56.601101 (+ 3558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:56.604083 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.604096 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.604119 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.604846 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.605247 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.605255 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.605261 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.605563 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.606290 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:56.606302 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:56.606931 (+ 629us) client_negotiation.cc:769] Sending connection context
0926 19:26:56.607328 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 19:26:56.608017 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":462,"threads_started":1}
I20260926 19:26:56.629968 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.616021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:54433 (local address 127.1.39.62:40879)
0926 19:26:56.616259 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:56.616268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:56.616821 (+ 553us) server_negotiation.cc:407] Connection header received
0926 19:26:56.616965 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:56.616973 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:56.617084 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:56.617215 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:56.618843 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.619755 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.621832 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.622267 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.622409 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.624223 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:56.624228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:56.624231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:56.624260 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:56.626945 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.627668 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.627675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.627678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.627796 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:56.628207 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:56.628214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:56.628220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:56.628699 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:56.628869 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:56.629419 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 19:26:56.629639 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:26:56.630084 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:56.616053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.2:54433)
0926 19:26:56.616711 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:56.616729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:56.616848 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:56.617275 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:56.617279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:56.617993 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:56.618694 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.618714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.619904 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.619909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.621691 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:56.621703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:56.622451 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:56.622456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:56.622616 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:56.623307 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:56.623323 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:56.624041 (+ 718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:56.627075 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.627082 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.627086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.627557 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.627908 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:56.627916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:56.627920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:56.628059 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:56.628873 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:56.628880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:56.629133 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 19:26:56.629299 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 19:26:56.629649 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":372,"threads_started":1}
I20260926 19:26:56.642987 2515 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:41411
I20260926 19:26:56.643028 2649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:41411 every 8 connection(s)
I20260926 19:26:56.644203 2515 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:26:56.646399 2650 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40879
I20260926 19:26:56.646593 2650 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:56.647068 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2515
I20260926 19:26:56.647226 2650 heartbeater.cc:511] Master 127.1.39.62:40879 requested a full tablet report, sending...
I20260926 19:26:56.647249 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:26:56.647686 1180 external_mini_cluster.cc:1428] Setting key 968cacc9468aa41af33b2616be7b253b
I20260926 19:26:56.648602 2304 ts_manager.cc:194] Registered new tserver with Master: 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:56.649767 2304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:54127
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:26:56.688158 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40879
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:57.013039 2654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:57.013429 2654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:57.013615 2654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:57.021991 2654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:57.022123 2654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:57.022295 2654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:26:57.022418 2654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:26:57.033102 2654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:40879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:57.034540 2654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:57.035871 2654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:57.049674 2660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:57.049641 2662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:57.049731 2659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:57.050547 2654 server_base.cc:1034] running on GCE node
I20260926 19:26:57.051172 2654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:57.052062 2654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:57.053380 2654 hybrid_clock.cc:648] HybridClock initialized: now 1790450817053266 us; error 64 us; skew 500 ppm
I20260926 19:26:57.053728 2654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:57.064265 2654 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:26:57.065836 2654 webserver.cc:460] Webserver started at http://127.1.39.3:39405/ using document root <none> and password file <none>
I20260926 19:26:57.066545 2654 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:57.066658 2654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:57.067000 2654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:57.069948 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "022b0d04e72c402895365902679a74b8"
format_stamp: "Formatted at 2026-09-26 19:26:57 on dist-test-slave-9wb0"
server_key: "69140cf3aa6274f18f7c90cb13afad60"
server_key_iv: "5a49e80eb1f43f94496ffb12baeb1c46"
server_key_version: "encryptionkey@0"
I20260926 19:26:57.070736 2654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "022b0d04e72c402895365902679a74b8"
format_stamp: "Formatted at 2026-09-26 19:26:57 on dist-test-slave-9wb0"
server_key: "69140cf3aa6274f18f7c90cb13afad60"
server_key_iv: "5a49e80eb1f43f94496ffb12baeb1c46"
server_key_version: "encryptionkey@0"
I20260926 19:26:57.076216 2654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:26:57.079707 2669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:57.080992 2654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:26:57.081173 2654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "022b0d04e72c402895365902679a74b8"
format_stamp: "Formatted at 2026-09-26 19:26:57 on dist-test-slave-9wb0"
server_key: "69140cf3aa6274f18f7c90cb13afad60"
server_key_iv: "5a49e80eb1f43f94496ffb12baeb1c46"
server_key_version: "encryptionkey@0"
I20260926 19:26:57.081467 2654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:57.095355 2654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:57.099434 2654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:57.099695 2654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:57.102201 2654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:26:57.102301 2654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:57.102423 2654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:26:57.102496 2654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:57.120908 2511 heartbeater.cc:503] Master 127.1.39.62:40879 was elected leader, sending a full tablet report...
I20260926 19:26:57.125326 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.102963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:47117 (local address 127.1.39.62:40879)
0926 19:26:57.103142 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.103147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.104824 (+ 1677us) server_negotiation.cc:407] Connection header received
0926 19:26:57.107238 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.107244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.107335 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.107446 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.109751 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.111039 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.113069 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.113642 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.113789 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.117758 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.117764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.117769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.117809 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.121046 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.122074 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.122080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.122083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.122249 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.122822 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.122826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.122830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.123201 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.124299 (+ 1098us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.124507 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.124820 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:26:57.125985 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.103783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.3:47117)
0926 19:26:57.104676 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.104718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.106927 (+ 2209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.107676 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.107683 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.108517 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:57.109517 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.109546 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.111251 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.111258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.112890 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.112901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.113855 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.113863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.114071 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.114733 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:57.114753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:57.117502 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:57.121251 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.121260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.121274 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.121921 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.122381 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.122390 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.122395 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.122661 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.123360 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:57.123370 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:57.123827 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.124125 (+ 298us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.124725 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":419,"threads_started":1}
I20260926 19:26:57.150436 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.133529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:49403 (local address 127.1.39.62:40879)
0926 19:26:57.133712 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.133717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.134513 (+ 796us) server_negotiation.cc:407] Connection header received
0926 19:26:57.134746 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.134754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.134870 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.134942 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.136461 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.137602 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.139598 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.140010 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.140151 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.142893 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.142902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.142910 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.142967 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.147055 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.147992 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.148000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.148004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.148138 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.148580 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.148587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.148593 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.148922 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.149073 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.149698 (+ 625us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.149993 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:26:57.150585 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.133744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.1.39.3:49403)
0926 19:26:57.134390 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.134413 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.134566 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.135146 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.135153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.135857 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:57.136273 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.136288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.137731 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.137739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.139448 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.139459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.140203 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.140211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.140411 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.141213 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:57.141240 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:57.142662 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:57.147247 (+ 4585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.147259 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.147263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.147852 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.148251 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.148258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.148263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.148444 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.149066 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:57.149069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:57.149342 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.149562 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.149918 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":348,"threads_started":1}
I20260926 19:26:57.151333 2654 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:40451
I20260926 19:26:57.152021 2788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:40451 every 8 connection(s)
I20260926 19:26:57.152815 2654 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:26:57.155064 2789 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40879
I20260926 19:26:57.155293 2789 heartbeater.cc:464] Registering TS with master...
I20260926 19:26:57.155411 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2654
I20260926 19:26:57.155623 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:26:57.155999 2789 heartbeater.cc:511] Master 127.1.39.62:40879 requested a full tablet report, sending...
I20260926 19:26:57.156029 1180 external_mini_cluster.cc:1428] Setting key 433e26d980485edba556bae13985874a
I20260926 19:26:57.157480 2303 ts_manager.cc:194] Registered new tserver with Master: 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451)
I20260926 19:26:57.158699 2303 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:47117
I20260926 19:26:57.159049 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:57.204790 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.190980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.1.39.62:40879)
0926 19:26:57.191207 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.191214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.191397 (+ 183us) server_negotiation.cc:407] Connection header received
0926 19:26:57.191629 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.191633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.191696 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.191804 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.193163 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.193945 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.195203 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.195611 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.195738 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.199611 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.199616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.199619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.199653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.202222 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.202867 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.202879 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.202882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.203008 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.203370 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.203377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.203380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.203651 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.203890 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.204188 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.204416 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:26:57.208706 2303 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:26:57.215653 2303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:26:57.238168 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.228095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34734 (local address 127.1.39.1:34543)
0926 19:26:57.230926 (+ 2831us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.230946 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.230975 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:26:57.231073 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.231079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.231310 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.231492 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.232318 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.233844 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.236195 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.237096 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.237244 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.237570 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.237708 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.237862 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2677,"thread_start_us":2552,"threads_started":1}
I20260926 19:26:57.238279 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.227875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34543 (local address 127.0.0.1:34734)
0926 19:26:57.230261 (+ 2386us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.230285 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.230426 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.231687 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.231691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.231712 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.232127 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.232139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.233990 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.233995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.236056 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.236064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.237282 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.237310 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.237488 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.237535 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.237673 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.237854 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2212,"mutex_wait_us":25,"thread_start_us":2025,"threads_started":1}
I20260926 19:26:57.239137 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.227671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40451 (local address 127.0.0.1:48954)
0926 19:26:57.228604 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.228659 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.228879 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.230505 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.230512 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.230543 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.231018 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.231032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.233340 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.233347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.235796 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.235809 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.238111 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.238117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.238520 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.238555 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.238639 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.238753 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":52,"thread_start_us":757,"threads_started":1}
I20260926 19:26:57.240130 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.228207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.1.39.3:40451)
0926 19:26:57.229819 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.229832 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.229874 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:26:57.229975 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.229985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.230263 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.231275 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.231627 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.233189 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.235956 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.237609 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.237736 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.237979 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.239662 (+ 1683us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.239817 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":1282,"threads_started":1}
I20260926 19:26:57.240703 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.230292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:41411 (local address 127.0.0.1:44210)
0926 19:26:57.232244 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.232261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.232354 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.233375 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.233380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.233407 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.233850 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.233860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.235615 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.235620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.237747 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.237760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.239983 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.239989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.240155 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.240185 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.240278 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.240414 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"thread_start_us":683,"threads_started":1}
I20260926 19:26:57.241441 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.230756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.1.39.2:41411)
0926 19:26:57.232815 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.232824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.232849 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:26:57.232930 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.232936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.233148 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.234056 (+ 908us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.234205 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.235415 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.237911 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.239794 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.239928 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.240122 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.240996 (+ 874us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.241150 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1913,"thread_start_us":1791,"threads_started":1}
I20260926 19:26:57.242427 2441 tablet_service.cc:1467] Processing CreateTablet for tablet 842a092fb0594f4ebf54a588334e532d (DEFAULT_TABLE table=test-table [id=a10b179e03ca4f10b7ef4357ad24520b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.243682 2441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842a092fb0594f4ebf54a588334e532d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.243330 2719 tablet_service.cc:1467] Processing CreateTablet for tablet 842a092fb0594f4ebf54a588334e532d (DEFAULT_TABLE table=test-table [id=a10b179e03ca4f10b7ef4357ad24520b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.244719 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842a092fb0594f4ebf54a588334e532d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.244480 2580 tablet_service.cc:1467] Processing CreateTablet for tablet 842a092fb0594f4ebf54a588334e532d (DEFAULT_TABLE table=test-table [id=a10b179e03ca4f10b7ef4357ad24520b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.245738 2580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 842a092fb0594f4ebf54a588334e532d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.258451 2808 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Bootstrap starting.
I20260926 19:26:57.261587 2808 tablet_bootstrap.cc:654] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.262049 2809 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Bootstrap starting.
I20260926 19:26:57.263048 2808 log.cc:824] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:57.264318 2810 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Bootstrap starting.
I20260926 19:26:57.265627 2809 tablet_bootstrap.cc:654] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.266491 2808 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: No bootstrap required, opened a new log
I20260926 19:26:57.266752 2808 ts_tablet_manager.cc:1397] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 19:26:57.266850 2809 log.cc:824] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:57.267947 2810 tablet_bootstrap.cc:654] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.269109 2810 log.cc:824] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Log is configured to *not* fsync() on all Append() calls
I20260926 19:26:57.269867 2809 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: No bootstrap required, opened a new log
I20260926 19:26:57.270146 2809 ts_tablet_manager.cc:1397] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:26:57.271895 2810 tablet_bootstrap.cc:492] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: No bootstrap required, opened a new log
I20260926 19:26:57.272154 2810 ts_tablet_manager.cc:1397] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260926 19:26:57.275161 2808 raft_consensus.cc:348] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.275507 2808 raft_consensus.cc:374] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.275628 2808 raft_consensus.cc:729] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27249121908d4ffb86138033c31bbaf2, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.276329 2808 consensus_queue.cc:260] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.278612 2808 ts_tablet_manager.cc:1428] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260926 19:26:57.280520 2809 raft_consensus.cc:348] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.280916 2809 raft_consensus.cc:374] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.281008 2809 raft_consensus.cc:729] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022b0d04e72c402895365902679a74b8, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.281977 2809 consensus_queue.cc:260] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.282464 2815 raft_consensus.cc:482] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:26:57.282761 2815 raft_consensus.cc:504] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.285152 2815 leader_election.cc:288] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543), 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:57.285598 2810 raft_consensus.cc:348] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.286077 2810 raft_consensus.cc:374] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.286172 2789 heartbeater.cc:503] Master 127.1.39.62:40879 was elected leader, sending a full tablet report...
I20260926 19:26:57.286178 2810 raft_consensus.cc:729] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4c66e24123465d969238c328ab2496, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.287524 2810 consensus_queue.cc:260] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.288465 2814 raft_consensus.cc:482] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:26:57.288743 2814 raft_consensus.cc:504] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.290508 2814 leader_election.cc:288] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451), 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:57.293892 2809 ts_tablet_manager.cc:1428] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Time spent starting tablet: real 0.023s user 0.006s sys 0.015s
I20260926 19:26:57.298532 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.285527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34543 (local address 127.1.39.3:54465)
0926 19:26:57.285862 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.285883 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.285999 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.286668 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.286672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.286693 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.286905 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.286917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.292801 (+ 5884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.292806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.295387 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.295406 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.297759 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.297763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.297946 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.297981 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.298089 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.298232 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":45}
I20260926 19:26:57.299268 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.285970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:54465 (local address 127.1.39.1:34543)
0926 19:26:57.286125 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.286130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.286166 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:26:57.286227 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.286231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.286309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.286406 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.287111 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.288215 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.295630 (+ 7415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.297415 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.297580 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.297672 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.298798 (+ 1126us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.298962 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 19:26:57.299755 2650 heartbeater.cc:503] Master 127.1.39.62:40879 was elected leader, sending a full tablet report...
I20260926 19:26:57.300209 2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27249121908d4ffb86138033c31bbaf2" is_pre_election: true
I20260926 19:26:57.300608 2466 raft_consensus.cc:2454] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 0.
I20260926 19:26:57.300738 2810 ts_tablet_manager.cc:1428] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Time spent starting tablet: real 0.028s user 0.009s sys 0.008s
I20260926 19:26:57.301663 2671 leader_election.cc:302] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022b0d04e72c402895365902679a74b8, 27249121908d4ffb86138033c31bbaf2; no voters:
I20260926 19:26:57.301944 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.286589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:48839 (local address 127.1.39.2:41411)
0926 19:26:57.286816 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.286823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.289138 (+ 2315us) server_negotiation.cc:407] Connection header received
0926 19:26:57.289975 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.289980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.290055 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.290170 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.290996 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.292084 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.299655 (+ 7571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.300959 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.301091 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.301207 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.301600 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.301725 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:26:57.302073 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.286422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:41411 (local address 127.1.39.3:48839)
0926 19:26:57.289020 (+ 2598us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.289049 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.289807 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.290321 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.290326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.290354 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.290809 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.290824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.292260 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.292265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.299486 (+ 7221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.299499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.301202 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.301208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.301423 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.301463 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.301561 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.301704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2421,"mutex_wait_us":29,"thread_start_us":2286,"threads_started":1}
I20260926 19:26:57.303158 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496" is_pre_election: true
I20260926 19:26:57.303720 2605 raft_consensus.cc:2454] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 0.
I20260926 19:26:57.304379 2816 raft_consensus.cc:2781] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:26:57.305552 2816 raft_consensus.cc:482] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:26:57.305696 2816 raft_consensus.cc:3037] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.308614 2816 raft_consensus.cc:504] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.309693 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.291572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:48837 (local address 127.1.39.2:41411)
0926 19:26:57.297184 (+ 5612us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.297192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.297887 (+ 695us) server_negotiation.cc:407] Connection header received
0926 19:26:57.298085 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.298090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.298157 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.298239 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.302498 (+ 4259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.304172 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.307371 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.308578 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.308698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.308824 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.309160 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.309320 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5448,"thread_start_us":5283,"threads_started":1}
I20260926 19:26:57.309734 2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27249121908d4ffb86138033c31bbaf2"
I20260926 19:26:57.309904 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.291781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:47673 (local address 127.1.39.3:40451)
0926 19:26:57.291960 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.291965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.299300 (+ 7335us) server_negotiation.cc:407] Connection header received
0926 19:26:57.300994 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.300999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.301077 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.301159 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:26:57.303121 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.304145 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.307323 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.308680 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.308824 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.308957 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.309539 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.309676 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:26:57.309931 2466 raft_consensus.cc:3037] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.310859 2819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.291109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40451 (local address 127.1.39.1:47673)
0926 19:26:57.299207 (+ 8098us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.299227 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.299351 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.302464 (+ 3113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.302472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.302499 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.302921 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.302935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.304318 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.304321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.307154 (+ 2833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.307169 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.309115 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.309121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.309359 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.309405 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.310055 (+ 650us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.310317 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7930,"mutex_wait_us":8,"thread_start_us":6261,"threads_started":1}
I20260926 19:26:57.311396 2820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.290919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:41411 (local address 127.1.39.1:48837)
0926 19:26:57.297751 (+ 6832us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.297779 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.297941 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.298500 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.298505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.298546 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:26:57.299203 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.299215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.304316 (+ 5101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.304321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.307252 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.307265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.308794 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.308802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.308988 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.309013 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.309125 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.310702 (+ 1577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6665,"mutex_wait_us":883,"thread_start_us":7966,"threads_started":1}
I20260926 19:26:57.312538 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496" is_pre_election: true
I20260926 19:26:57.312774 2605 raft_consensus.cc:2454] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27249121908d4ffb86138033c31bbaf2 in term 0.
I20260926 19:26:57.313364 2392 leader_election.cc:302] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4c66e24123465d969238c328ab2496, 27249121908d4ffb86138033c31bbaf2; no voters:
I20260926 19:26:57.314666 2466 raft_consensus.cc:2454] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 1.
I20260926 19:26:57.314829 2814 raft_consensus.cc:2741] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 19:26:57.315147 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022b0d04e72c402895365902679a74b8" is_pre_election: true
I20260926 19:26:57.315083 2671 leader_election.cc:302] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022b0d04e72c402895365902679a74b8, 27249121908d4ffb86138033c31bbaf2; no voters:
I20260926 19:26:57.315693 2744 raft_consensus.cc:2379] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 27249121908d4ffb86138033c31bbaf2 in current term 1: Already voted for candidate 022b0d04e72c402895365902679a74b8 in this term.
I20260926 19:26:57.316120 2815 raft_consensus.cc:2781] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:57.317655 2815 raft_consensus.cc:686] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 LEADER]: Becoming Leader. State: Replica: 022b0d04e72c402895365902679a74b8, State: Running, Role: LEADER
I20260926 19:26:57.318184 2815 consensus_queue.cc:237] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } }
I20260926 19:26:57.321856 2816 leader_election.cc:288] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 election: Requested vote from peers 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543), 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:57.322454 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "842a092fb0594f4ebf54a588334e532d" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496"
I20260926 19:26:57.322690 2605 raft_consensus.cc:3037] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.326757 2605 raft_consensus.cc:2454] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 1.
I20260926 19:26:57.326531 2303 catalog_manager.cc:5495] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022b0d04e72c402895365902679a74b8 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "022b0d04e72c402895365902679a74b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } health_report { overall_health: UNKNOWN } } }
W20260926 19:26:57.362805 2512 tablet.cc:2369] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:26:57.371814 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.366276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.1.39.3:40451)
0926 19:26:57.366456 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.366464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.366491 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:26:57.366853 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.366857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.366947 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.367095 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:26:57.367450 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.368200 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.369557 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.369945 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.370100 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.370726 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.371198 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.371349 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.371467 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.371541 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:26:57.380168 2466 raft_consensus.cc:1261] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Refusing update from remote peer 022b0d04e72c402895365902679a74b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.380174 2605 raft_consensus.cc:1261] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Refusing update from remote peer 022b0d04e72c402895365902679a74b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.381016 2815 consensus_queue.cc:1035] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.381322 2816 consensus_queue.cc:1035] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.393554 2826 mvcc.cc:205] Tried to move back new op lower bound from 7333686547979513856 to 7333686547741851648. Current Snapshot: MvccSnapshot[applied={T|T < 7333686547979513856}]
I20260926 19:26:57.409762 2304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:26:57.410002 2304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:26:57.413554 2304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Sending DeleteTablet for 3 replicas of tablet 842a092fb0594f4ebf54a588334e532d
I20260926 19:26:57.414741 2719 tablet_service.cc:1514] Processing DeleteTablet for tablet 842a092fb0594f4ebf54a588334e532d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48954
I20260926 19:26:57.416075 2304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:26:57.416723 2304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:26:57.416734 2580 tablet_service.cc:1514] Processing DeleteTablet for tablet 842a092fb0594f4ebf54a588334e532d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260926 19:26:57.415163 2441 tablet_service.cc:1514] Processing DeleteTablet for tablet 842a092fb0594f4ebf54a588334e532d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:34734
I20260926 19:26:57.425400 2718 tablet_service.cc:1467] Processing CreateTablet for tablet 870b23eebb074828a1c2efaff5e4333f (DEFAULT_TABLE table=test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.426051 2718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870b23eebb074828a1c2efaff5e4333f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.426013 2440 tablet_service.cc:1467] Processing CreateTablet for tablet 870b23eebb074828a1c2efaff5e4333f (DEFAULT_TABLE table=test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.426544 2440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870b23eebb074828a1c2efaff5e4333f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.425216 2579 tablet_service.cc:1467] Processing CreateTablet for tablet 870b23eebb074828a1c2efaff5e4333f (DEFAULT_TABLE table=test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:26:57.429822 2579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 870b23eebb074828a1c2efaff5e4333f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.429836 2836 tablet_replica.cc:330] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: stopping tablet replica
I20260926 19:26:57.430344 2836 raft_consensus.cc:2229] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:26:57.432973 2835 tablet_replica.cc:330] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: stopping tablet replica
I20260926 19:26:57.433467 2835 raft_consensus.cc:2229] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:26:57.434046 2809 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Bootstrap starting.
I20260926 19:26:57.434047 2835 raft_consensus.cc:2258] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.435061 2836 raft_consensus.cc:2258] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.435877 2809 tablet_bootstrap.cc:654] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.440996 2808 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Bootstrap starting.
I20260926 19:26:57.442152 2810 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Bootstrap starting.
I20260926 19:26:57.442541 2808 tablet_bootstrap.cc:654] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.444281 2810 tablet_bootstrap.cc:654] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.454577 2834 tablet_replica.cc:330] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: stopping tablet replica
I20260926 19:26:57.454608 2809 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: No bootstrap required, opened a new log
I20260926 19:26:57.454795 2809 ts_tablet_manager.cc:1397] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.001s
I20260926 19:26:57.455134 2834 raft_consensus.cc:2229] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:26:57.455703 2809 raft_consensus.cc:348] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.455976 2834 raft_consensus.cc:2258] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.455981 2809 raft_consensus.cc:374] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.456162 2809 raft_consensus.cc:729] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022b0d04e72c402895365902679a74b8, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.456449 2809 consensus_queue.cc:260] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.457763 2809 ts_tablet_manager.cc:1428] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 19:26:57.458099 2835 ts_tablet_manager.cc:1905] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.459182 2836 ts_tablet_manager.cc:1905] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.460247 2810 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: No bootstrap required, opened a new log
I20260926 19:26:57.460538 2810 ts_tablet_manager.cc:1397] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20260926 19:26:57.461475 2810 raft_consensus.cc:348] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.461753 2810 raft_consensus.cc:374] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.461894 2810 raft_consensus.cc:729] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4c66e24123465d969238c328ab2496, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.462282 2816 raft_consensus.cc:482] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:26:57.462278 2810 consensus_queue.cc:260] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.462441 2816 raft_consensus.cc:504] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.463095 2816 leader_election.cc:288] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543), 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:57.464699 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870b23eebb074828a1c2efaff5e4333f" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496" is_pre_election: true
I20260926 19:26:57.464910 2605 raft_consensus.cc:2454] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 0.
I20260926 19:26:57.465240 2810 ts_tablet_manager.cc:1428] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 19:26:57.465323 2671 leader_election.cc:302] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022b0d04e72c402895365902679a74b8, 0d4c66e24123465d969238c328ab2496; no voters:
I20260926 19:26:57.465615 2816 raft_consensus.cc:2781] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:26:57.465728 2816 raft_consensus.cc:482] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:26:57.465817 2816 raft_consensus.cc:3037] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.466527 2834 ts_tablet_manager.cc:1905] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.464011 2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870b23eebb074828a1c2efaff5e4333f" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27249121908d4ffb86138033c31bbaf2" is_pre_election: true
I20260926 19:26:57.470824 2808 tablet_bootstrap.cc:492] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: No bootstrap required, opened a new log
I20260926 19:26:57.471012 2835 ts_tablet_manager.cc:1918] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:26:57.471045 2808 ts_tablet_manager.cc:1397] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.004s
I20260926 19:26:57.471179 2835 log.cc:1196] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/842a092fb0594f4ebf54a588334e532d
I20260926 19:26:57.471726 2835 ts_tablet_manager.cc:1939] T 842a092fb0594f4ebf54a588334e532d P 0d4c66e24123465d969238c328ab2496: Deleting consensus metadata
I20260926 19:26:57.472018 2808 raft_consensus.cc:348] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.472242 2808 raft_consensus.cc:374] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.472357 2808 raft_consensus.cc:729] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27249121908d4ffb86138033c31bbaf2, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.472631 2808 consensus_queue.cc:260] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.475407 2808 ts_tablet_manager.cc:1428] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 19:26:57.475464 2836 ts_tablet_manager.cc:1918] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:26:57.475620 2836 log.cc:1196] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/842a092fb0594f4ebf54a588334e532d
I20260926 19:26:57.476169 2836 ts_tablet_manager.cc:1939] T 842a092fb0594f4ebf54a588334e532d P 27249121908d4ffb86138033c31bbaf2: Deleting consensus metadata
I20260926 19:26:57.476714 2289 catalog_manager.cc:4841] TS 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411): tablet 842a092fb0594f4ebf54a588334e532d (table test-table [id=a10b179e03ca4f10b7ef4357ad24520b]) successfully deleted
W20260926 19:26:57.477447 2671 leader_election.cc:341] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543): Illegal state: must be running to vote when last-logged opid is not known
I20260926 19:26:57.477756 2289 catalog_manager.cc:4841] TS 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543): tablet 842a092fb0594f4ebf54a588334e532d (table test-table [id=a10b179e03ca4f10b7ef4357ad24520b]) successfully deleted
I20260926 19:26:57.477456 2816 raft_consensus.cc:504] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.478973 2466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870b23eebb074828a1c2efaff5e4333f" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27249121908d4ffb86138033c31bbaf2"
I20260926 19:26:57.479135 2816 leader_election.cc:288] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 election: Requested vote from peers 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543), 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411)
I20260926 19:26:57.479175 2466 raft_consensus.cc:3037] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.480281 2834 ts_tablet_manager.cc:1918] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:26:57.480436 2834 log.cc:1196] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/842a092fb0594f4ebf54a588334e532d
I20260926 19:26:57.480996 2834 ts_tablet_manager.cc:1939] T 842a092fb0594f4ebf54a588334e532d P 022b0d04e72c402895365902679a74b8: Deleting consensus metadata
I20260926 19:26:57.482053 2466 raft_consensus.cc:2454] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 1.
I20260926 19:26:57.482599 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "870b23eebb074828a1c2efaff5e4333f" candidate_uuid: "022b0d04e72c402895365902679a74b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496"
I20260926 19:26:57.482813 2605 raft_consensus.cc:3037] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.485757 2605 raft_consensus.cc:2454] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 022b0d04e72c402895365902679a74b8 in term 1.
I20260926 19:26:57.486254 2671 leader_election.cc:302] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 022b0d04e72c402895365902679a74b8, 0d4c66e24123465d969238c328ab2496; no voters:
I20260926 19:26:57.486549 2816 raft_consensus.cc:2781] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:57.486706 2816 raft_consensus.cc:686] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 1 LEADER]: Becoming Leader. State: Replica: 022b0d04e72c402895365902679a74b8, State: Running, Role: LEADER
I20260926 19:26:57.487073 2291 catalog_manager.cc:4841] TS 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451): tablet 842a092fb0594f4ebf54a588334e532d (table test-table [id=a10b179e03ca4f10b7ef4357ad24520b]) successfully deleted
I20260926 19:26:57.486967 2816 consensus_queue.cc:237] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.489578 2304 catalog_manager.cc:5495] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 022b0d04e72c402895365902679a74b8 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "022b0d04e72c402895365902679a74b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } health_report { overall_health: HEALTHY } } }
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450815, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:57.547791 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.531711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40879 (local address 127.0.0.1:45870)
0926 19:26:57.532356 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.532382 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.532504 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.532861 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.532864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.533613 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:26:57.533882 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.533891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.535272 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.535276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.536246 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.536255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.537247 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.537251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.537434 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.538379 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:26:57.538396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:26:57.541139 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:26:57.545087 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.545097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.545102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.545669 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.546176 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:26:57.546180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:26:57.546182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:26:57.546306 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:26:57.546989 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:26:57.546996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:26:57.547134 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.547270 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.547486 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":372,"threads_started":1}
I20260926 19:26:57.548079 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.531705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.1.39.62:40879)
0926 19:26:57.531903 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.531911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.532476 (+ 565us) server_negotiation.cc:407] Connection header received
0926 19:26:57.532623 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.532629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.532706 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.532849 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.534040 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.535156 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.536378 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.537073 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.537199 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.541384 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.541393 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.541398 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.541439 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.544924 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.545792 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.545798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.545802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.546049 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.546491 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.546496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.546500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.546860 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.547010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.547439 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.547686 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:26:57.553471 2304 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45870:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:26:57.554812 2304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:26:57.561942 2440 tablet_service.cc:1467] Processing CreateTablet for tablet d9b45a27b9fc4fe89cc40ae685b26c88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c02eedbd3b4be69195418aaa970c5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:26:57.561990 2579 tablet_service.cc:1467] Processing CreateTablet for tablet d9b45a27b9fc4fe89cc40ae685b26c88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c02eedbd3b4be69195418aaa970c5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:26:57.562463 2440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b45a27b9fc4fe89cc40ae685b26c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.562482 2579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b45a27b9fc4fe89cc40ae685b26c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.566195 2719 tablet_service.cc:1467] Processing CreateTablet for tablet d9b45a27b9fc4fe89cc40ae685b26c88 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=89c02eedbd3b4be69195418aaa970c5c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:26:57.566753 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b45a27b9fc4fe89cc40ae685b26c88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:57.570531 2810 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: Bootstrap starting.
I20260926 19:26:57.571839 2808 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: Bootstrap starting.
I20260926 19:26:57.572362 2810 tablet_bootstrap.cc:654] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.574150 2808 tablet_bootstrap.cc:654] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.574738 2809 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: Bootstrap starting.
I20260926 19:26:57.576419 2809 tablet_bootstrap.cc:654] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:57.582111 2809 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: No bootstrap required, opened a new log
I20260926 19:26:57.582288 2809 ts_tablet_manager.cc:1397] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260926 19:26:57.582851 2808 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: No bootstrap required, opened a new log
I20260926 19:26:57.582849 2810 tablet_bootstrap.cc:492] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: No bootstrap required, opened a new log
I20260926 19:26:57.583029 2808 ts_tablet_manager.cc:1397] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 19:26:57.583127 2809 raft_consensus.cc:348] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.583366 2809 raft_consensus.cc:374] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.583446 2809 raft_consensus.cc:729] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022b0d04e72c402895365902679a74b8, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.583678 2809 consensus_queue.cc:260] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.583768 2808 raft_consensus.cc:348] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.584048 2808 raft_consensus.cc:374] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.584138 2808 raft_consensus.cc:729] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27249121908d4ffb86138033c31bbaf2, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.584251 2809 ts_tablet_manager.cc:1428] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 19:26:57.584405 2808 consensus_queue.cc:260] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.584632 2810 ts_tablet_manager.cc:1397] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20260926 19:26:57.584614 2816 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.584935 2816 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.585099 2808 ts_tablet_manager.cc:1428] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:26:57.585754 2810 raft_consensus.cc:348] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.586082 2810 raft_consensus.cc:374] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:57.586233 2810 raft_consensus.cc:729] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4c66e24123465d969238c328ab2496, State: Initialized, Role: FOLLOWER
I20260926 19:26:57.586336 2814 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.586659 2814 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.586668 2810 consensus_queue.cc:260] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.587759 2810 ts_tablet_manager.cc:1428] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:26:57.587512 2818 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.587957 2818 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.644136 2814 raft_consensus.cc:482] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:26:57.644336 2814 raft_consensus.cc:504] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.644989 2814 leader_election.cc:288] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411), 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451)
I20260926 19:26:57.645540 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b45a27b9fc4fe89cc40ae685b26c88" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496" is_pre_election: true
I20260926 19:26:57.645803 2605 raft_consensus.cc:2454] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27249121908d4ffb86138033c31bbaf2 in term 0.
I20260926 19:26:57.645771 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b45a27b9fc4fe89cc40ae685b26c88" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022b0d04e72c402895365902679a74b8" is_pre_election: true
I20260926 19:26:57.646020 2744 raft_consensus.cc:2454] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27249121908d4ffb86138033c31bbaf2 in term 0.
I20260926 19:26:57.646220 2392 leader_election.cc:302] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4c66e24123465d969238c328ab2496, 27249121908d4ffb86138033c31bbaf2; no voters:
I20260926 19:26:57.646485 2814 raft_consensus.cc:2781] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:26:57.646620 2814 raft_consensus.cc:482] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:26:57.646700 2814 raft_consensus.cc:3037] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 0 FOLLOWER]: Advancing to term 1
W20260926 19:26:57.648630 2651 tablet.cc:2369] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:26:57.649850 2814 raft_consensus.cc:504] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.651312 2605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b45a27b9fc4fe89cc40ae685b26c88" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4c66e24123465d969238c328ab2496"
I20260926 19:26:57.651532 2605 raft_consensus.cc:3037] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.651750 2744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b45a27b9fc4fe89cc40ae685b26c88" candidate_uuid: "27249121908d4ffb86138033c31bbaf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "022b0d04e72c402895365902679a74b8"
I20260926 19:26:57.651949 2744 raft_consensus.cc:3037] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:57.654465 2605 raft_consensus.cc:2454] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27249121908d4ffb86138033c31bbaf2 in term 1.
I20260926 19:26:57.654938 2392 leader_election.cc:302] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4c66e24123465d969238c328ab2496, 27249121908d4ffb86138033c31bbaf2; no voters:
I20260926 19:26:57.655193 2744 raft_consensus.cc:2454] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27249121908d4ffb86138033c31bbaf2 in term 1.
I20260926 19:26:57.655792 2814 leader_election.cc:288] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [CANDIDATE]: Term 1 election: Requested vote from peers 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411), 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451)
I20260926 19:26:57.655987 2814 raft_consensus.cc:2781] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:57.656124 2814 raft_consensus.cc:686] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [term 1 LEADER]: Becoming Leader. State: Replica: 27249121908d4ffb86138033c31bbaf2, State: Running, Role: LEADER
W20260926 19:26:57.656155 2790 tablet.cc:2369] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:26:57.656510 2814 consensus_queue.cc:237] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } }
I20260926 19:26:57.667025 2850 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: TxnStatusTablet state changed. Reason: New leader 27249121908d4ffb86138033c31bbaf2. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.667568 2850 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:26:57.667956 2303 catalog_manager.cc:5495] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27249121908d4ffb86138033c31bbaf2 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } health_report { overall_health: UNKNOWN } } }
I20260926 19:26:57.672969 2852 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:26:57.673197 2852 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:26:57.712868 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.704796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34543 (local address 127.0.0.1:34742)
0926 19:26:57.705152 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.705173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.705336 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.706185 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.706191 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.706205 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:26:57.706393 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.706404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.707895 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.707901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.709539 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.709553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.710531 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.710538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.710735 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.710801 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.712253 (+ 1452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.712276 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.712400 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.712563 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 19:26:57.712875 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.704648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.1.39.1:34543)
0926 19:26:57.704828 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.704835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.705569 (+ 734us) server_negotiation.cc:407] Connection header received
0926 19:26:57.705677 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.705683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.705828 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.706001 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:26:57.706555 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.707754 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.709688 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.710366 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.710502 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.711015 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.712135 (+ 1120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.712276 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.712446 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.712528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:26:57.717342 2605 raft_consensus.cc:1261] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Refusing update from remote peer 27249121908d4ffb86138033c31bbaf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.717366 2744 raft_consensus.cc:1261] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Refusing update from remote peer 27249121908d4ffb86138033c31bbaf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.718084 2850 consensus_queue.cc:1035] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.719296 2814 consensus_queue.cc:1035] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.723016 2818 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: TxnStatusTablet state changed. Reason: New leader 27249121908d4ffb86138033c31bbaf2. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.723343 2818 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.726037 2814 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.726352 2814 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:26:57.728261 2818 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.728538 2818 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 0d4c66e24123465d969238c328ab2496: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.728721 2827 mvcc.cc:205] Tried to move back new op lower bound from 7333686549362462720 to 7333686549125677056. Current Snapshot: MvccSnapshot[applied={T|T < 7333686549362462720}]
I20260926 19:26:57.733497 2816 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: TxnStatusTablet state changed. Reason: New leader 27249121908d4ffb86138033c31bbaf2. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.733829 2816 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.736088 2814 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.737856 2814 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 27249121908d4ffb86138033c31bbaf2: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:26:57.739490 2825 mvcc.cc:205] Tried to move back new op lower bound from 7333686549362462720 to 7333686549125677056. Current Snapshot: MvccSnapshot[applied={T|T < 7333686549362462720 or (T in {7333686549362462720})}]
I20260926 19:26:57.740566 2816 tablet_replica.cc:439] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "27249121908d4ffb86138033c31bbaf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 } } peers { permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 } } peers { permanent_uuid: "022b0d04e72c402895365902679a74b8" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40451 } } }
I20260926 19:26:57.740860 2816 tablet_replica.cc:442] T d9b45a27b9fc4fe89cc40ae685b26c88 P 022b0d04e72c402895365902679a74b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:26:57.760944 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.752372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:36647 (local address 127.1.39.1:34543)
0926 19:26:57.752565 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.752572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.753443 (+ 871us) server_negotiation.cc:407] Connection header received
0926 19:26:57.753671 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.753677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.753780 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.753888 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:26:57.754645 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.756234 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.758652 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.759380 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.759624 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.760174 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.760297 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.760391 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.760555 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.760638 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:26:57.761158 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.752421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34543 (local address 127.1.39.3:36647)
0926 19:26:57.753283 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.753333 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.753507 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.753946 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.753951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.753966 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:26:57.754459 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.754473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.756383 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.756389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.758481 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.758498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.759611 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.759621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.759902 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.759988 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.760410 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.760424 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.760520 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.760673 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":496,"threads_started":1}
I20260926 19:26:57.771907 2605 raft_consensus.cc:1261] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Refusing update from remote peer 022b0d04e72c402895365902679a74b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.772147 2466 raft_consensus.cc:1261] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Refusing update from remote peer 022b0d04e72c402895365902679a74b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:26:57.772666 2816 consensus_queue.cc:1035] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4c66e24123465d969238c328ab2496" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.773193 2816 consensus_queue.cc:1035] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27249121908d4ffb86138033c31bbaf2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:26:57.833565 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.821529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:39493 (local address 127.1.39.3:40451)
0926 19:26:57.821706 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.821713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.822315 (+ 602us) server_negotiation.cc:407] Connection header received
0926 19:26:57.822646 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.822652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.822741 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.822804 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:26:57.823479 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.824887 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.827745 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.828538 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.828655 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.829382 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.829522 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.832774 (+ 3252us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.833009 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.833098 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:26:57.834371 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.821184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40451 (local address 127.1.39.1:39493)
0926 19:26:57.822210 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 19:26:57.822232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:26:57.822447 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:26:57.823022 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:26:57.823028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:26:57.823042 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:26:57.823313 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.823324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.825091 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.825096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.827522 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:26:57.827538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.828842 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.828848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:26:57.829048 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.829136 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:26:57.832809 (+ 3673us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:26:57.832823 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 19:26:57.833822 (+ 999us) client_negotiation.cc:240] Negotiation successful
0926 19:26:57.834012 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":2793,"threads_started":1}
I20260926 19:26:57.883914 2303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:26:57.884202 2303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:26:57.888224 2303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40b1197f9f8a4ce381757dfbb12b8e9a: Sending DeleteTablet for 3 replicas of tablet 870b23eebb074828a1c2efaff5e4333f
I20260926 19:26:57.889315 2579 tablet_service.cc:1514] Processing DeleteTablet for tablet 870b23eebb074828a1c2efaff5e4333f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:44210
I20260926 19:26:57.889381 2719 tablet_service.cc:1514] Processing DeleteTablet for tablet 870b23eebb074828a1c2efaff5e4333f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48954
I20260926 19:26:57.889605 2835 tablet_replica.cc:330] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: stopping tablet replica
I20260926 19:26:57.889675 2834 tablet_replica.cc:330] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: stopping tablet replica
I20260926 19:26:57.889789 2835 raft_consensus.cc:2229] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:26:57.889858 2834 raft_consensus.cc:2229] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:26:57.889961 2835 raft_consensus.cc:2258] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.890221 2834 raft_consensus.cc:2258] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.890645 2440 tablet_service.cc:1514] Processing DeleteTablet for tablet 870b23eebb074828a1c2efaff5e4333f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:26:57 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:34734
I20260926 19:26:57.890897 2836 tablet_replica.cc:330] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: stopping tablet replica
I20260926 19:26:57.891078 2836 raft_consensus.cc:2229] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:26:57.891263 2836 raft_consensus.cc:2258] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:26:57.891388 2834 ts_tablet_manager.cc:1905] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.892782 2836 ts_tablet_manager.cc:1905] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.894094 2835 ts_tablet_manager.cc:1905] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:26:57.899057 2834 ts_tablet_manager.cc:1918] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:26:57.899206 2834 log.cc:1196] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/870b23eebb074828a1c2efaff5e4333f
I20260926 19:26:57.899611 2834 ts_tablet_manager.cc:1939] T 870b23eebb074828a1c2efaff5e4333f P 022b0d04e72c402895365902679a74b8: Deleting consensus metadata
I20260926 19:26:57.900688 2291 catalog_manager.cc:4841] TS 022b0d04e72c402895365902679a74b8 (127.1.39.3:40451): tablet 870b23eebb074828a1c2efaff5e4333f (table test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]) successfully deleted
I20260926 19:26:57.903537 2835 ts_tablet_manager.cc:1918] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:26:57.903690 2835 log.cc:1196] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/870b23eebb074828a1c2efaff5e4333f
I20260926 19:26:57.904147 2835 ts_tablet_manager.cc:1939] T 870b23eebb074828a1c2efaff5e4333f P 0d4c66e24123465d969238c328ab2496: Deleting consensus metadata
I20260926 19:26:57.905390 2289 catalog_manager.cc:4841] TS 0d4c66e24123465d969238c328ab2496 (127.1.39.2:41411): tablet 870b23eebb074828a1c2efaff5e4333f (table test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]) successfully deleted
I20260926 19:26:57.909541 2836 ts_tablet_manager.cc:1918] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:26:57.909823 2836 log.cc:1196] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/870b23eebb074828a1c2efaff5e4333f
I20260926 19:26:57.910331 2836 ts_tablet_manager.cc:1939] T 870b23eebb074828a1c2efaff5e4333f P 27249121908d4ffb86138033c31bbaf2: Deleting consensus metadata
I20260926 19:26:57.911377 2289 catalog_manager.cc:4841] TS 27249121908d4ffb86138033c31bbaf2 (127.1.39.1:34543): tablet 870b23eebb074828a1c2efaff5e4333f (table test-table [id=77d7cd3716ed4472a5e15fef991b6d1f]) successfully deleted
Sep 26 19:26:57 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:26:57.931456 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.906175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.1.39.1:34543)
0926 19:26:57.906361 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.906369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.912172 (+ 5803us) server_negotiation.cc:407] Connection header received
0926 19:26:57.912344 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.912349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.912435 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.912547 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.914437 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.916013 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.917851 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.918620 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.918749 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.924022 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.924038 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.924059 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.924105 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.928212 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.929102 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.929115 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.929119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.929363 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.929824 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.929829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.929833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.930292 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.930617 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.930884 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.931105 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 19:26:57.932646 2506 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34744
I20260926 19:26:57.958930 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.939787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.1.39.62:40879)
0926 19:26:57.939985 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.939992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.941727 (+ 1735us) server_negotiation.cc:407] Connection header received
0926 19:26:57.942033 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.942039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.942127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.942203 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.944533 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.945716 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.947505 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.948087 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.948204 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.951892 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.951899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.951904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.951943 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.955702 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.956550 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.956556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.956560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.956721 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.957177 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.957182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.957186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.957582 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.957814 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.958163 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.958379 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 19:26:57.959510 2303 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45874
I20260926 19:26:57.961177 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2375
I20260926 19:26:57.979254 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:57.962961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.1.39.1:34543)
0926 19:26:57.963165 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:57.963171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:57.963642 (+ 471us) server_negotiation.cc:407] Connection header received
0926 19:26:57.963802 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:57.963808 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:57.963897 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:57.963997 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:57.965456 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.966660 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.968402 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:57.969075 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:57.969197 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:57.971997 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:57.972003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:57.972009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:57.972052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:57.976209 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.977135 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.977140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.977143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.977365 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:57.977776 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:57.977781 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:57.977785 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:57.978123 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:57.978357 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:57.978656 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 19:26:57.978886 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 19:26:57.979825 2506 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34754
W20260926 19:26:57.980408 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:26:57.980595 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2375
I20260926 19:26:58.000052 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2515
Sep 26 19:26:58 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:26:58.014812 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:58.000766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.1.39.2:41411)
0926 19:26:58.001217 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:58.001221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:58.001236 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:26:58.001287 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:58.001344 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:58.001429 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:58.001518 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:58.002613 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:58.003304 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:58.004491 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:58.004892 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:58.005027 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:58.008902 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:58.008916 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:58.008932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:58.008970 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:58.011848 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:58.012575 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:58.012582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:58.012585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:58.012722 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:58.013165 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:58.013175 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:58.013178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:58.013664 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:58.013905 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:58.014142 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 19:26:58.014406 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
W20260926 19:26:58.015609 2645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44226
W20260926 19:26:58.016249 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2515: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:26:58.016451 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2515
I20260926 19:26:58.033946 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2654
Sep 26 19:26:58 dist-test-slave-9wb0 krb5kdc[2259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:26:58.048484 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:58.034733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48970 (local address 127.1.39.3:40451)
0926 19:26:58.034938 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:58.034944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:58.034963 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:26:58.035096 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:58.035102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:58.035179 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:58.035287 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:58.036523 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:58.037254 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:58.038477 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:58.038905 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:58.039035 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:58.043366 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:58.043379 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:58.043395 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:58.043424 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:58.046254 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:58.046869 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:58.046876 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:58.046878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:58.047001 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:58.047283 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:58.047285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:58.047287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:58.047624 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:58.047867 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:58.048096 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 19:26:58.048252 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 19:26:58.049145 2784 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48970
W20260926 19:26:58.049809 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2654: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:26:58.049963 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2654
I20260926 19:26:58.070947 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2272
I20260926 19:26:58.072088 2344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:26:58.187922 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2272
2026-09-26T19:26:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4435 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:26:58 dist-test-slave-9wb0 krb5kdc[2874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:26:58 dist-test-slave-9wb0 krb5kdc[2874](info): set up 2 sockets
Sep 26 19:26:58 dist-test-slave-9wb0 krb5kdc[2874](info): commencing operation
krb5kdc: starting...
W20260926 19:26:59.285447 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:26:59 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:26:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:26:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:26:59.470404 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42069
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:26:59.785274 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:26:59.785584 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:26:59.785766 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:26:59.792938 2887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:26:59.793053 2887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:26:59.793136 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:26:59.793256 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:26:59.793332 2887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:26:59.793439 2887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:26:59.803102 2887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42069
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:26:59.804435 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:26:59.805830 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:26:59.817621 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:59.817701 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:26:59.817655 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:26:59.818495 2887 server_base.cc:1034] running on GCE node
I20260926 19:26:59.819237 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:26:59.820593 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:26:59.821851 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790450819821789 us; error 51 us; skew 500 ppm
I20260926 19:26:59.822201 2887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:26:59 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:26:59.825270 2887 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:26:59.826864 2887 webserver.cc:460] Webserver started at http://127.1.39.62:35881/ using document root <none> and password file <none>
I20260926 19:26:59.827560 2887 fs_manager.cc:359] Metadata directory not provided
I20260926 19:26:59.827693 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:26:59.828004 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:26:59.830996 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "7fc4b154637a42b9afd124bb2872dc63"
format_stamp: "Formatted at 2026-09-26 19:26:59 on dist-test-slave-9wb0"
server_key: "0da9595913c47eb1a79817ba3053e9f3"
server_key_iv: "da1a0b97145eec90c502bd271197321d"
server_key_version: "encryptionkey@0"
I20260926 19:26:59.831776 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "7fc4b154637a42b9afd124bb2872dc63"
format_stamp: "Formatted at 2026-09-26 19:26:59 on dist-test-slave-9wb0"
server_key: "0da9595913c47eb1a79817ba3053e9f3"
server_key_iv: "da1a0b97145eec90c502bd271197321d"
server_key_version: "encryptionkey@0"
I20260926 19:26:59.836969 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 19:26:59.840433 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:26:59.841903 2887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:26:59.842087 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "7fc4b154637a42b9afd124bb2872dc63"
format_stamp: "Formatted at 2026-09-26 19:26:59 on dist-test-slave-9wb0"
server_key: "0da9595913c47eb1a79817ba3053e9f3"
server_key_iv: "da1a0b97145eec90c502bd271197321d"
server_key_version: "encryptionkey@0"
I20260926 19:26:59.842308 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:26:59.862902 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:26:59.867254 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:26:59.867549 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:26:59.889038 2887 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42069
I20260926 19:26:59.889045 2963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42069 every 8 connection(s)
I20260926 19:26:59.890460 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:26:59.894562 2964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:26:59.897128 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2887
I20260926 19:26:59.897370 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:26:59.898209 1180 external_mini_cluster.cc:1428] Setting key 2783737339ee549b8db23d901a79c3d9
I20260926 19:26:59.907492 2964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Bootstrap starting.
I20260926 19:26:59.911520 2964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Neither blocks nor log segments found. Creating new log.
I20260926 19:26:59.912899 2964 log.cc:824] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:26:59 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:26:59.916309 2964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: No bootstrap required, opened a new log
I20260926 19:26:59.923161 2967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:26:59.900343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.1.39.62:42069)
0926 19:26:59.901500 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0926 19:26:59.901512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:26:59.901559 (+ 47us) server_negotiation.cc:407] Connection header received
0926 19:26:59.903274 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:26:59.903299 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:26:59.903753 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:26:59.904133 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:26:59.906002 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:59.907322 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:59.909074 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:26:59.909843 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:26:59.909981 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:26:59.914877 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:26:59.914887 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:26:59.914906 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:26:59.914948 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:26:59.918850 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:59.919475 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:59.919488 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:59.919497 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:59.919697 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:26:59.920028 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:26:59.920035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:26:59.920042 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:26:59.920747 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:26:59.921043 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 19:26:59.921319 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 19:26:59.921766 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":528,"threads_started":1}
I20260926 19:26:59.926904 2964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } }
I20260926 19:26:59.927238 2964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:26:59.927313 2964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fc4b154637a42b9afd124bb2872dc63, State: Initialized, Role: FOLLOWER
I20260926 19:26:59.927860 2964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } }
I20260926 19:26:59.928058 2964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:26:59.928158 2964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:26:59.928274 2964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:26:59.930687 2964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } }
I20260926 19:26:59.931147 2964 leader_election.cc:302] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fc4b154637a42b9afd124bb2872dc63; no voters:
I20260926 19:26:59.931797 2964 leader_election.cc:288] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:26:59.931946 2969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:26:59.932715 2969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [term 1 LEADER]: Becoming Leader. State: Replica: 7fc4b154637a42b9afd124bb2872dc63, State: Running, Role: LEADER
I20260926 19:26:59.933285 2969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } }
I20260926 19:26:59.934077 2964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:26:59.936846 2971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fc4b154637a42b9afd124bb2872dc63. Latest consensus state: current_term: 1 leader_uuid: "7fc4b154637a42b9afd124bb2872dc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } } }
I20260926 19:26:59.937265 2971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:59.936920 2970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fc4b154637a42b9afd124bb2872dc63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fc4b154637a42b9afd124bb2872dc63" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42069 } } }
I20260926 19:26:59.938352 2970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63 [sys.catalog]: This master's current role is: LEADER
I20260926 19:26:59.938892 2978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:26:59.944124 2978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:26:59.953518 2978 catalog_manager.cc:1347] Generated new cluster ID: a1410ea9d8c8400f9653b67c59438962
I20260926 19:26:59.953648 2978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:26:59.962280 2978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:26:59.963514 2978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:26:59.978997 2978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Generated new TSK 0
I20260926 19:26:59.979852 2978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:00.037388 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42069
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:00.365929 2990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:00.366199 2990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:00.366297 2990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:00.373541 2990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:00.373652 2990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:00.373739 2990 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:27:00.373790 2990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:00.373849 2990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:00.383356 2990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:00.384681 2990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:00.386018 2990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:00.398383 2998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:00.398448 2995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:00.398469 2996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:00.399291 2990 server_base.cc:1034] running on GCE node
I20260926 19:27:00.399883 2990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:00.400761 2990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:00.401976 2990 hybrid_clock.cc:648] HybridClock initialized: now 1790450820401936 us; error 39 us; skew 500 ppm
I20260926 19:27:00.402280 2990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:00 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:00.405469 2990 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:00.406950 2990 webserver.cc:460] Webserver started at http://127.1.39.1:37593/ using document root <none> and password file <none>
I20260926 19:27:00.407625 2990 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:00.407769 2990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:00.408075 2990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:00.411139 2990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "4e63721b603245bebcd2f029db93f9c8"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "a3d93dd7f0e3ee66dfcf30cab4847ebb"
server_key_iv: "a7b43475aeef1f5605bf2359b0b1d986"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.411890 2990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "4e63721b603245bebcd2f029db93f9c8"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "a3d93dd7f0e3ee66dfcf30cab4847ebb"
server_key_iv: "a7b43475aeef1f5605bf2359b0b1d986"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.417068 2990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:27:00.420533 3005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:00.422158 2990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:00.422340 2990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "4e63721b603245bebcd2f029db93f9c8"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "a3d93dd7f0e3ee66dfcf30cab4847ebb"
server_key_iv: "a7b43475aeef1f5605bf2359b0b1d986"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.422544 2990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:00.439492 2990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:00.443531 2990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:00.443815 2990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:00.446820 2990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:00.446957 2990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:00.447114 2990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:00.447232 2990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:00 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:00.471868 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.447230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:33087 (local address 127.1.39.62:42069)
0926 19:27:00.450046 (+ 2816us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:00.450051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:00.450070 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:00.452324 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:00.452332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:00.452425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:00.452588 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:00.455158 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.456268 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.459055 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.459711 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.459863 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.464539 (+ 4676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:00.464545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:00.464548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:00.464581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:00.468038 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.469142 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.469148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.469152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.469336 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.469905 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.469910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.469914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.470211 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:00.470368 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:00.471275 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 19:27:00.471499 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2716,"thread_start_us":2621,"threads_started":1}
I20260926 19:27:00.473260 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.448029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.1:33087)
0926 19:27:00.449055 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:00.449107 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:00.451933 (+ 2826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:00.452913 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:00.452929 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:00.453923 (+ 994us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:00.454911 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.454947 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.456475 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.456483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.458846 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.458869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.459989 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.459997 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.460394 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.461283 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:00.461366 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:00.464283 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:00.468237 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.468246 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.468262 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.468940 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.469474 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.469484 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.469489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.469774 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.470343 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:00.470351 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:00.470836 (+ 485us) client_negotiation.cc:769] Sending connection context
0926 19:27:00.471189 (+ 353us) client_negotiation.cc:240] Negotiation successful
0926 19:27:00.471734 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":440,"threads_started":1}
I20260926 19:27:00.495718 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.480210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:59087 (local address 127.1.39.62:42069)
0926 19:27:00.480359 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:00.480363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:00.481018 (+ 655us) server_negotiation.cc:407] Connection header received
0926 19:27:00.481214 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:00.481219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:00.481340 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:00.481462 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:00.482331 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.483046 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.485033 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.485800 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.485912 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.489256 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:00.489262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:00.489267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:00.489343 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:00.492675 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.493578 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.493585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.493589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.493753 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.494271 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.494279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.494285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.494625 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:00.494767 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:00.495169 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 19:27:00.495377 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:27:00.495850 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.480333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.1:59087)
0926 19:27:00.480912 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:00.480927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:00.481089 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:00.481487 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:00.481491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:00.481917 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:00.482197 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.482205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.483168 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.483176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.484848 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.484865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.486266 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.486274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.486531 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.487310 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:00.487342 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:00.489047 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:00.492831 (+ 3784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.492837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.492841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.493449 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.493902 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.493912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.493919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.494138 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.494753 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:00.494757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:00.494930 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 19:27:00.495079 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 19:27:00.495394 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":339,"threads_started":1}
I20260926 19:27:00.502039 2990 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:40729
I20260926 19:27:00.502089 3125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:40729 every 8 connection(s)
I20260926 19:27:00.503223 2990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:00.505061 3126 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42069
I20260926 19:27:00.505280 3126 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:00.505947 3126 heartbeater.cc:511] Master 127.1.39.62:42069 requested a full tablet report, sending...
I20260926 19:27:00.507023 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 2990
I20260926 19:27:00.507243 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:00.508038 1180 external_mini_cluster.cc:1428] Setting key 89f317fddac9c44cf5e51ae09eae5491
I20260926 19:27:00.508141 2919 ts_manager.cc:194] Registered new tserver with Master: 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729)
I20260926 19:27:00.510241 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:33087
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:00.548306 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42069
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:00.869626 3130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:00.869956 3130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:00.870148 3130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:00.878010 3130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:00.878173 3130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:00.878311 3130 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:27:00.878399 3130 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:00.878532 3130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:00.888362 3130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:00.889997 3130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:00.891649 3130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:00.906109 3136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:00.906149 3138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:00.906809 3135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:00.907418 3130 server_base.cc:1034] running on GCE node
I20260926 19:27:00.908144 3130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:00.909219 3130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:00.910494 3130 hybrid_clock.cc:648] HybridClock initialized: now 1790450820910432 us; error 54 us; skew 500 ppm
I20260926 19:27:00.910851 3130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:00 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:00.913985 3130 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:00.915341 3130 webserver.cc:460] Webserver started at http://127.1.39.2:39335/ using document root <none> and password file <none>
I20260926 19:27:00.916018 3130 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:00.916150 3130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:00.916453 3130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:00.919268 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "5aa4f831cd994adb97442cfd3de6610b"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "0c4e9a053267dd84c837deff9fc94db6"
server_key_iv: "97da8db1540b3fcd32b34bc81cf3fe8b"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.919994 3130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "5aa4f831cd994adb97442cfd3de6610b"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "0c4e9a053267dd84c837deff9fc94db6"
server_key_iv: "97da8db1540b3fcd32b34bc81cf3fe8b"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.925361 3130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:27:00.928898 3145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:00.930212 3130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:00.930375 3130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "5aa4f831cd994adb97442cfd3de6610b"
format_stamp: "Formatted at 2026-09-26 19:27:00 on dist-test-slave-9wb0"
server_key: "0c4e9a053267dd84c837deff9fc94db6"
server_key_iv: "97da8db1540b3fcd32b34bc81cf3fe8b"
server_key_version: "encryptionkey@0"
I20260926 19:27:00.930548 3130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:00.955228 3130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:00.959097 3130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:00.959396 3130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:00.962054 3130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:00.962167 3130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:00.962332 3130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:00.962437 3130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:00 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450820, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:00.985839 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.962576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:47403 (local address 127.1.39.62:42069)
0926 19:27:00.962765 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:00.962773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:00.964195 (+ 1422us) server_negotiation.cc:407] Connection header received
0926 19:27:00.966280 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:00.966286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:00.966380 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:00.966490 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:00.968611 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.970026 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.972643 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.973060 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.973218 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.978295 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:00.978304 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:00.978312 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:00.978366 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:00.982399 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.983432 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.983441 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.983447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.983618 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:00.984089 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:00.984096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:00.984100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:00.984389 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:00.984555 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:00.985241 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 19:27:00.985480 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:27:00.986873 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:00.963222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.2:47403)
0926 19:27:00.964027 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:00.964064 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:00.965987 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:00.966727 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:00.966734 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:00.967515 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:00.968369 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.968399 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.970228 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.970236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.972455 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:00.972471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:00.973370 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:00.973381 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:00.973758 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:00.974704 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:00.974726 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:00.978048 (+ 3322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:00.982618 (+ 4570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.982629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.982657 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.983296 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.983744 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:00.983754 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:00.983758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:00.983940 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:00.984518 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:00.984526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:00.984869 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 19:27:00.985139 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 19:27:00.985697 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":369,"threads_started":1}
I20260926 19:27:01.018054 3130 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:35435
I20260926 19:27:01.018142 3264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:35435 every 8 connection(s)
I20260926 19:27:01.018889 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.000815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.2:54741)
0926 19:27:01.001550 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.001576 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.001772 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.002437 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.002446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.003248 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:01.003839 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.003861 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.005407 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.005416 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.007667 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.007686 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.008688 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.008696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.008885 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.009707 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:01.009740 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:01.011230 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:01.015809 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.015814 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.015817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.016344 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.016836 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.016841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.016845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.017004 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.017851 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:01.017859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:01.018110 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.018293 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.018559 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":410,"threads_started":1}
I20260926 19:27:01.018977 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.000528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:54741 (local address 127.1.39.62:42069)
0926 19:27:01.000707 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.000711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.001669 (+ 958us) server_negotiation.cc:407] Connection header received
0926 19:27:01.001978 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.001987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.002084 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.002210 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:01.004050 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.005223 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.007834 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.008487 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.008594 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.011416 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:01.011424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:01.011432 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:01.011484 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:01.015616 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.016519 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.016524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.016528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.016689 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.017156 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.017164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.017171 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.017698 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:01.017880 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.018439 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.018670 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:27:01.019717 3130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:01.022073 3265 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42069
I20260926 19:27:01.022316 3265 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:01.023138 3265 heartbeater.cc:511] Master 127.1.39.62:42069 requested a full tablet report, sending...
I20260926 19:27:01.024449 2919 ts_manager.cc:194] Registered new tserver with Master: 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435)
I20260926 19:27:01.025274 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:47403
I20260926 19:27:01.026225 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3130
I20260926 19:27:01.026383 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:01.026852 1180 external_mini_cluster.cc:1428] Setting key 2664b02f184df7aee21df4d5b5e3679c
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:01.068662 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42069
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:01.382920 3269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:01.383190 3269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:01.383353 3269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:01.391248 3269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:01.391366 3269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:01.391449 3269 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 19:27:01.391503 3269 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:01.391566 3269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:01.401173 3269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:42069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:01.402485 3269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:01.403838 3269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:01.417079 3275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:01.417157 3277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:01.417133 3274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:01.418787 3269 server_base.cc:1034] running on GCE node
I20260926 19:27:01.419392 3269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:01.420341 3269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:01.421540 3269 hybrid_clock.cc:648] HybridClock initialized: now 1790450821421489 us; error 40 us; skew 500 ppm
I20260926 19:27:01.421876 3269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:01 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450821, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:01.425031 3269 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:01.426776 3269 webserver.cc:460] Webserver started at http://127.1.39.3:46833/ using document root <none> and password file <none>
I20260926 19:27:01.427613 3269 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:01.427757 3269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:01.428071 3269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:01.431032 3269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "209bef2e40244773b02b8eff4e4e5505"
format_stamp: "Formatted at 2026-09-26 19:27:01 on dist-test-slave-9wb0"
server_key: "f7db9f130d88d38759dabe45b62a443c"
server_key_iv: "03043d83344b31531020c0a419c7cd13"
server_key_version: "encryptionkey@0"
I20260926 19:27:01.431799 3269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "209bef2e40244773b02b8eff4e4e5505"
format_stamp: "Formatted at 2026-09-26 19:27:01 on dist-test-slave-9wb0"
server_key: "f7db9f130d88d38759dabe45b62a443c"
server_key_iv: "03043d83344b31531020c0a419c7cd13"
server_key_version: "encryptionkey@0"
I20260926 19:27:01.437131 3269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:27:01.440662 3284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:01.442056 3269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:01.442246 3269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "209bef2e40244773b02b8eff4e4e5505"
format_stamp: "Formatted at 2026-09-26 19:27:01 on dist-test-slave-9wb0"
server_key: "f7db9f130d88d38759dabe45b62a443c"
server_key_iv: "03043d83344b31531020c0a419c7cd13"
server_key_version: "encryptionkey@0"
I20260926 19:27:01.442456 3269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:01.460881 3269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:01.464538 3269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:01.464830 3269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:01.467540 3269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:01.467674 3269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:01.467840 3269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:01.467960 3269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:01 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450821, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:01.490993 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.467973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:44921 (local address 127.1.39.62:42069)
0926 19:27:01.468143 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.468147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.469518 (+ 1371us) server_negotiation.cc:407] Connection header received
0926 19:27:01.471639 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.471648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.471741 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.471865 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:01.474046 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.475152 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.477874 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.478566 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.478731 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.484498 (+ 5767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:01.484504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:01.484507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:01.484539 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:01.487217 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.488315 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.488319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.488322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.488444 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.489030 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.489036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.489042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.489467 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:01.489636 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.490428 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.490644 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:27:01.491983 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.468579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.3:44921)
0926 19:27:01.469375 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.469410 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.471280 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.472155 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.472166 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.472996 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:01.473840 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.473868 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.475325 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.475334 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.477688 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.477699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.478837 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.478845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.479106 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.479917 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:01.479947 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:01.484158 (+ 4211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:01.487369 (+ 3211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.487383 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.487406 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.488182 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.488555 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.488563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.488569 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.488842 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.489610 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:01.489617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:01.489951 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.490291 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.490806 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":318,"threads_started":1}
I20260926 19:27:01.510380 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.497619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:33919 (local address 127.1.39.62:42069)
0926 19:27:01.497802 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.497808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.498729 (+ 921us) server_negotiation.cc:407] Connection header received
0926 19:27:01.498939 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.498944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.499045 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.499150 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:01.500503 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.501384 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.502720 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.503151 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.503294 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.505006 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:01.505012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:01.505017 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:01.505080 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:01.507819 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.508446 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.508452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.508456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.508579 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.508915 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.508922 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.508925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.509194 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:01.509380 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.509791 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.510000 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:27:01.510458 3343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.497729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.1.39.3:33919)
0926 19:27:01.498609 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.498634 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.498802 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.499205 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.499214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.499885 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:01.500348 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.500364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.501513 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.501519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.502579 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.502590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.503332 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.503339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.503519 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.504129 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:01.504146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:01.504818 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:01.507950 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.507957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.507961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.508330 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.508685 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.508691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.508694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.508814 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.509333 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:01.509338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:01.509517 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.509675 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.509993 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":533,"threads_started":1}
I20260926 19:27:01.513068 3126 heartbeater.cc:503] Master 127.1.39.62:42069 was elected leader, sending a full tablet report...
I20260926 19:27:01.530982 3269 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:41277
I20260926 19:27:01.531075 3403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:41277 every 8 connection(s)
I20260926 19:27:01.532244 3269 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:01.534408 3404 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42069
I20260926 19:27:01.534664 3404 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:01.535288 3404 heartbeater.cc:511] Master 127.1.39.62:42069 requested a full tablet report, sending...
I20260926 19:27:01.536516 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3269
I20260926 19:27:01.536597 2919 ts_manager.cc:194] Registered new tserver with Master: 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:01.536739 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:01.537217 1180 external_mini_cluster.cc:1428] Setting key ddf1b53927a2f9ad73f0946f9c006e16
I20260926 19:27:01.537601 2919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:44921
I20260926 19:27:01.540151 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:27:01 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:27:01 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:01.586951 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.573816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.1.39.62:42069)
0926 19:27:01.574069 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.574075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.574301 (+ 226us) server_negotiation.cc:407] Connection header received
0926 19:27:01.574465 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.574469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.574540 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.574659 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:01.576209 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.576939 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.578198 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.578611 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.578717 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.582265 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:01.582270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:01.582273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:01.582302 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:01.584835 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.585394 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.585397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.585399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.585517 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.585809 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.585812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.585814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.586042 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:01.586220 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.586418 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.586567 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 19:27:01.590744 2919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:01.597580 2919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:01.621389 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.609602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35435 (local address 127.0.0.1:45428)
0926 19:27:01.610394 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.610436 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.610605 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.611724 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.611734 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.611776 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:01.612254 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.612271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.614658 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.614667 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.617747 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.617766 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.620062 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.620074 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.620373 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.620449 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.620583 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.620831 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
I20260926 19:27:01.622684 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.609866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.1.39.2:35435)
0926 19:27:01.610423 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.610434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.610539 (+ 105us) server_negotiation.cc:407] Connection header received
0926 19:27:01.610850 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.610860 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.611187 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.611412 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:01.612491 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.614436 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.617995 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.619780 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.621944 (+ 2164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.622195 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.622264 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.622423 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":265,"threads_started":1}
I20260926 19:27:01.624064 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.610694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:41277 (local address 127.0.0.1:45274)
0926 19:27:01.612865 (+ 2171us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.612884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.612985 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.614403 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.614409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.614435 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:01.615054 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.615070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.617789 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.617796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.620698 (+ 2902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.620718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.623344 (+ 2626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.623350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.623535 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.623572 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.623644 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.623727 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":1869,"threads_started":1}
I20260926 19:27:01.625048 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.610970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.1.39.3:41277)
0926 19:27:01.613696 (+ 2726us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.613707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.613747 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:27:01.613859 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.613870 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.614154 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.615304 (+ 1150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:01.615684 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.617606 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.620964 (+ 3358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.622761 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.624279 (+ 1518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.624512 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.624575 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.624743 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2495,"thread_start_us":2272,"threads_started":1}
I20260926 19:27:01.626981 3195 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1315c0c4694f808e166447186d50ee (DEFAULT_TABLE table=test-table [id=805dc46fe49a4057854300f381418b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:01.628489 3195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1315c0c4694f808e166447186d50ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.628204 3334 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1315c0c4694f808e166447186d50ee (DEFAULT_TABLE table=test-table [id=805dc46fe49a4057854300f381418b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:01.629339 3417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.613569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.0.0.1:59198)
0926 19:27:01.621651 (+ 8082us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.621667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.621751 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.623309 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.623314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.623341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:01.623517 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.623536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.625041 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.625048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.626812 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.626837 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.628572 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.628581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.628760 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.628795 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.628898 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.629024 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7999,"thread_start_us":8988,"threads_started":1}
I20260926 19:27:01.629606 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1315c0c4694f808e166447186d50ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.630033 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.613217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.1.39.1:40729)
0926 19:27:01.622591 (+ 9374us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.622611 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.622641 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:27:01.622742 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.622750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.623101 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.623701 (+ 600us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:01.623893 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.624894 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.627012 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.628401 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.628549 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.628749 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.628959 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.629762 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8942,"thread_start_us":8807,"threads_started":1}
I20260926 19:27:01.633332 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 4b1315c0c4694f808e166447186d50ee (DEFAULT_TABLE table=test-table [id=805dc46fe49a4057854300f381418b51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:01.634632 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b1315c0c4694f808e166447186d50ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.641216 3423 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Bootstrap starting.
I20260926 19:27:01.643657 3424 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Bootstrap starting.
I20260926 19:27:01.645015 3423 tablet_bootstrap.cc:654] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.646421 3423 log.cc:824] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:01.647274 3424 tablet_bootstrap.cc:654] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.648524 3424 log.cc:824] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:01.649504 3426 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Bootstrap starting.
I20260926 19:27:01.649639 3423 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: No bootstrap required, opened a new log
I20260926 19:27:01.649899 3423 ts_tablet_manager.cc:1397] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 19:27:01.651268 3424 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: No bootstrap required, opened a new log
I20260926 19:27:01.651512 3424 ts_tablet_manager.cc:1397] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 19:27:01.652828 3426 tablet_bootstrap.cc:654] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.654045 3426 log.cc:824] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:01.657095 3426 tablet_bootstrap.cc:492] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: No bootstrap required, opened a new log
I20260926 19:27:01.657385 3426 ts_tablet_manager.cc:1397] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260926 19:27:01.658977 3424 raft_consensus.cc:348] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.659343 3424 raft_consensus.cc:374] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.659441 3424 raft_consensus.cc:729] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.660091 3424 consensus_queue.cc:260] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.660161 3423 raft_consensus.cc:348] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.660545 3423 raft_consensus.cc:374] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.660626 3423 raft_consensus.cc:729] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.661258 3423 consensus_queue.cc:260] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.662776 3265 heartbeater.cc:503] Master 127.1.39.62:42069 was elected leader, sending a full tablet report...
I20260926 19:27:01.663280 3423 ts_tablet_manager.cc:1428] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260926 19:27:01.664179 3404 heartbeater.cc:503] Master 127.1.39.62:42069 was elected leader, sending a full tablet report...
I20260926 19:27:01.664770 3424 ts_tablet_manager.cc:1428] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260926 19:27:01.667243 3426 raft_consensus.cc:348] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.667658 3426 raft_consensus.cc:374] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.667763 3426 raft_consensus.cc:729] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.668465 3426 consensus_queue.cc:260] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.670265 3426 ts_tablet_manager.cc:1428] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260926 19:27:01.676551 3431 raft_consensus.cc:482] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:01.676801 3431 raft_consensus.cc:504] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.678258 3431 leader_election.cc:288] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:01.684481 3418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.678552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:45459 (local address 127.1.39.2:35435)
0926 19:27:01.678750 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.678758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.679322 (+ 564us) server_negotiation.cc:407] Connection header received
0926 19:27:01.679535 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.679543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.679662 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.679753 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:01.680301 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.681021 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.682825 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.683537 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.683720 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.683822 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.684106 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.684229 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:27:01.684525 3432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.678613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35435 (local address 127.1.39.1:45459)
0926 19:27:01.679195 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.679210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.679370 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.679831 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.679835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.679862 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:01.680174 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.680183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.681149 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.681156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.682629 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.682641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.683745 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.683751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.683936 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.683966 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.684064 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.684190 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":334,"threads_started":1}
I20260926 19:27:01.685413 3420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.679529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57669 (local address 127.1.39.3:41277)
0926 19:27:01.679728 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.679735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.679990 (+ 255us) server_negotiation.cc:407] Connection header received
0926 19:27:01.680180 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.680187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.680245 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.680333 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:01.680830 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.681584 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.683158 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.683998 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.684133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.684226 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.685015 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.685134 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:27:01.685463 3433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.679317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:41277 (local address 127.1.39.1:57669)
0926 19:27:01.679891 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.679908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.680055 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.680355 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.680359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.680381 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:01.680692 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.680700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.681714 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.681718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.683017 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.683029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.684383 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.684389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.684851 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.684885 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.684990 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.685117 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":315,"threads_started":1}
I20260926 19:27:01.685539 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1315c0c4694f808e166447186d50ee" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b" is_pre_election: true
I20260926 19:27:01.686239 3220 raft_consensus.cc:2454] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:01.687096 3007 leader_election.cc:302] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e63721b603245bebcd2f029db93f9c8, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:01.687521 3431 raft_consensus.cc:2781] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:01.687681 3431 raft_consensus.cc:482] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:01.687795 3431 raft_consensus.cc:3037] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.687752 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1315c0c4694f808e166447186d50ee" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505" is_pre_election: true
I20260926 19:27:01.688282 3359 raft_consensus.cc:2454] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:01.690462 3431 raft_consensus.cc:504] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.690943 3431 leader_election.cc:288] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Requested vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:01.691380 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1315c0c4694f808e166447186d50ee" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b"
I20260926 19:27:01.691452 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b1315c0c4694f808e166447186d50ee" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505"
I20260926 19:27:01.691591 3220 raft_consensus.cc:3037] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.691641 3359 raft_consensus.cc:3037] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.694796 3220 raft_consensus.cc:2454] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:01.694796 3359 raft_consensus.cc:2454] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:01.695199 3007 leader_election.cc:302] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 4e63721b603245bebcd2f029db93f9c8; no voters:
I20260926 19:27:01.695443 3431 raft_consensus.cc:2781] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:01.695892 3431 raft_consensus.cc:686] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Running, Role: LEADER
I20260926 19:27:01.696331 3431 consensus_queue.cc:237] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } }
I20260926 19:27:01.701116 2919 catalog_manager.cc:5495] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } health_report { overall_health: HEALTHY } } }
W20260926 19:27:01.755543 3127 tablet.cc:2369] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 19:27:01 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450819, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:01.772275 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.759722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42069 (local address 127.0.0.1:40922)
0926 19:27:01.760349 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:01.760365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:01.760495 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:01.760780 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:01.760783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:01.761447 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:01.761723 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.761731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.762730 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.762734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.763658 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:01.763667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.764380 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.764384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:01.764501 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.765250 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:01.765266 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:01.767371 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:01.770017 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.770026 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.770032 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.770429 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.770862 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:01.770866 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:01.770868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:01.770994 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:01.771437 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:01.771442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:01.771586 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 19:27:01.771708 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 19:27:01.771887 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":371,"threads_started":1}
I20260926 19:27:01.772287 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:01.759722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.1.39.62:42069)
0926 19:27:01.759899 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:01.759903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:01.760489 (+ 586us) server_negotiation.cc:407] Connection header received
0926 19:27:01.760599 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:01.760603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:01.760664 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:01.760742 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:01.761868 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.762616 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.763798 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:01.764219 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:01.764316 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:01.767526 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:01.767531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:01.767535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:01.767563 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:01.769909 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.770540 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.770545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.770547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.770730 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:01.771088 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:01.771091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:01.771093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:01.771334 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:01.771440 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:01.771873 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 19:27:01.772020 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 19:27:01.772384 3266 tablet.cc:2369] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:01.777266 2919 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:40922:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:27:01.778312 2919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260926 19:27:01.783840 3405 tablet.cc:2369] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:01.784449 3056 tablet_service.cc:1467] Processing CreateTablet for tablet 33fcc4d8a4464be6b253b9041efe9bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aded7e8c8eaf4fa4948c81568c32fae1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:01.784623 3195 tablet_service.cc:1467] Processing CreateTablet for tablet 33fcc4d8a4464be6b253b9041efe9bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aded7e8c8eaf4fa4948c81568c32fae1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:01.785468 3195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fcc4d8a4464be6b253b9041efe9bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.785468 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fcc4d8a4464be6b253b9041efe9bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.785511 3334 tablet_service.cc:1467] Processing CreateTablet for tablet 33fcc4d8a4464be6b253b9041efe9bc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aded7e8c8eaf4fa4948c81568c32fae1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:01.786258 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fcc4d8a4464be6b253b9041efe9bc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:01.791492 3424 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: Bootstrap starting.
I20260926 19:27:01.792557 3423 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: Bootstrap starting.
I20260926 19:27:01.792809 3426 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: Bootstrap starting.
I20260926 19:27:01.793222 3424 tablet_bootstrap.cc:654] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.794063 3423 tablet_bootstrap.cc:654] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.794708 3426 tablet_bootstrap.cc:654] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:01.802356 3423 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: No bootstrap required, opened a new log
I20260926 19:27:01.802479 3426 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: No bootstrap required, opened a new log
I20260926 19:27:01.802561 3423 ts_tablet_manager.cc:1397] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260926 19:27:01.802646 3426 ts_tablet_manager.cc:1397] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 19:27:01.803176 3424 tablet_bootstrap.cc:492] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: No bootstrap required, opened a new log
I20260926 19:27:01.803316 3424 ts_tablet_manager.cc:1397] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260926 19:27:01.803372 3423 raft_consensus.cc:348] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.803643 3423 raft_consensus.cc:374] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.803610 3426 raft_consensus.cc:348] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.803741 3423 raft_consensus.cc:729] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.803822 3426 raft_consensus.cc:374] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.803916 3426 raft_consensus.cc:729] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.803961 3423 consensus_queue.cc:260] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.804067 3424 raft_consensus.cc:348] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.804145 3426 consensus_queue.cc:260] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.804306 3424 raft_consensus.cc:374] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:01.804394 3424 raft_consensus.cc:729] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Initialized, Role: FOLLOWER
I20260926 19:27:01.804550 3423 ts_tablet_manager.cc:1428] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 19:27:01.804653 3424 consensus_queue.cc:260] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.804734 3431 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:01.804934 3430 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:01.805178 3431 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:01.805194 3424 ts_tablet_manager.cc:1428] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 19:27:01.805271 3430 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:01.806067 3426 ts_tablet_manager.cc:1428] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:01.806039 3429 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:01.806373 3429 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:01.983877 3431 raft_consensus.cc:482] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:01.984122 3431 raft_consensus.cc:504] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.984733 3431 leader_election.cc:288] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:01.985323 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fcc4d8a4464be6b253b9041efe9bc4" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b" is_pre_election: true
I20260926 19:27:01.985360 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fcc4d8a4464be6b253b9041efe9bc4" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505" is_pre_election: true
I20260926 19:27:01.985590 3220 raft_consensus.cc:2454] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:01.985612 3359 raft_consensus.cc:2454] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:01.986004 3007 leader_election.cc:302] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 4e63721b603245bebcd2f029db93f9c8; no voters:
I20260926 19:27:01.986248 3431 raft_consensus.cc:2781] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:01.986330 3431 raft_consensus.cc:482] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:01.986431 3431 raft_consensus.cc:3037] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.988368 3431 raft_consensus.cc:504] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.988915 3431 leader_election.cc:288] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Requested vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:01.989362 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fcc4d8a4464be6b253b9041efe9bc4" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b"
I20260926 19:27:01.989410 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fcc4d8a4464be6b253b9041efe9bc4" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505"
I20260926 19:27:01.989593 3220 raft_consensus.cc:3037] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.989656 3359 raft_consensus.cc:3037] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:01.991557 3359 raft_consensus.cc:2454] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:01.991567 3220 raft_consensus.cc:2454] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:01.991938 3007 leader_election.cc:302] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e63721b603245bebcd2f029db93f9c8, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:01.992173 3431 raft_consensus.cc:2781] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:01.992378 3431 raft_consensus.cc:686] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Running, Role: LEADER
I20260926 19:27:01.992636 3431 consensus_queue.cc:237] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:01.994127 3434 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: TxnStatusTablet state changed. Reason: New leader 4e63721b603245bebcd2f029db93f9c8. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:01.994547 3434 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:01.995270 3446 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:27:01.995571 3446 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:27:01.995829 2918 catalog_manager.cc:5495] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:02.040616 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.034872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.1.39.1:40729)
0926 19:27:02.035069 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:02.035075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:02.035312 (+ 237us) server_negotiation.cc:407] Connection header received
0926 19:27:02.035589 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:02.035592 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:02.035691 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:02.035831 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:02.036162 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.037023 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.038520 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.038939 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.039059 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.039519 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:02.040009 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:02.040164 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:02.040242 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 19:27:02.040314 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:27:02.040676 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.034817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.0.0.1:59200)
0926 19:27:02.035226 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:02.035241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:02.035390 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:02.035847 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:02.035851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:02.035861 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:02.036038 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.036046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.037148 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.037154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.038393 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.038404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.039084 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.039090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.039271 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.039319 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:02.040124 (+ 805us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:02.040143 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 19:27:02.040224 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 19:27:02.040361 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260926 19:27:02.048161 3220 raft_consensus.cc:1261] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Refusing update from remote peer 4e63721b603245bebcd2f029db93f9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:02.048177 3359 raft_consensus.cc:1261] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Refusing update from remote peer 4e63721b603245bebcd2f029db93f9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:02.049393 3434 consensus_queue.cc:1035] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:02.049741 3431 consensus_queue.cc:1035] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:02.058118 3430 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: TxnStatusTablet state changed. Reason: New leader 4e63721b603245bebcd2f029db93f9c8. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.058414 3430 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:02.061174 3450 mvcc.cc:205] Tried to move back new op lower bound from 7333686567099002880 to 7333686566883880960. Current Snapshot: MvccSnapshot[applied={T|T < 7333686567099002880}]
I20260926 19:27:02.064822 3430 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.065060 3430 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 5aa4f831cd994adb97442cfd3de6610b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:02.065964 3431 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.066280 3431 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:02.070838 3429 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: TxnStatusTablet state changed. Reason: New leader 4e63721b603245bebcd2f029db93f9c8. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.071103 3429 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:02.074440 3434 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.074694 3434 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 4e63721b603245bebcd2f029db93f9c8: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:02.088414 3429 tablet_replica.cc:439] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } } }
I20260926 19:27:02.088686 3429 tablet_replica.cc:442] T 33fcc4d8a4464be6b253b9041efe9bc4 P 209bef2e40244773b02b8eff4e4e5505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:02.097482 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.091994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.1.39.1:40729)
0926 19:27:02.092166 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:02.092171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:02.092643 (+ 472us) server_negotiation.cc:407] Connection header received
0926 19:27:02.092841 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:02.092846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:02.092958 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:02.093064 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:02.093571 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.094430 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.095770 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.096200 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.096339 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.096742 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:02.096846 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:02.096951 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:02.097093 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 19:27:02.097165 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:27:02.102586 3456 consensus_queue.cc:1035] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:02.110625 3434 consensus_queue.cc:1035] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:02.116845 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.109239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:42693 (local address 127.1.39.1:40729)
0926 19:27:02.109451 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:02.109460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:02.111755 (+ 2295us) server_negotiation.cc:407] Connection header received
0926 19:27:02.111997 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:02.112003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:02.112068 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:02.112134 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:02.112831 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.113713 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.115129 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.115567 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.115711 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.116105 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:02.116202 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:02.116293 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:02.116425 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 19:27:02.116485 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:27:02.117153 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.109084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.1.39.1:42693)
0926 19:27:02.111671 (+ 2587us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:02.111692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:02.111857 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:02.112355 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:02.112359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:02.112373 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:02.112684 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.112691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.113838 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.113846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.114985 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.114995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.115752 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.115758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.115876 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.115921 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:02.116292 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:02.116306 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 19:27:02.116404 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:27:02.116526 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2406,"thread_start_us":3052,"threads_started":1}
I20260926 19:27:02.177937 2918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:02.178200 2918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:02.181269 2918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Sending DeleteTablet for 3 replicas of tablet 4b1315c0c4694f808e166447186d50ee
I20260926 19:27:02.182341 3195 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1315c0c4694f808e166447186d50ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:02 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45428
I20260926 19:27:02.182386 3334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1315c0c4694f808e166447186d50ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:02 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260926 19:27:02.183700 2918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:02.184310 2918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:02.189119 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b1315c0c4694f808e166447186d50ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:02 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260926 19:27:02.192642 3194 tablet_service.cc:1467] Processing CreateTablet for tablet 41f853e5dd6f4edca374acd67312ed9f (DEFAULT_TABLE table=test-table [id=1f0f8ca2fc0746ee8706c368393bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:02.193192 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f853e5dd6f4edca374acd67312ed9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:02.195118 3468 tablet_replica.cc:330] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: stopping tablet replica
I20260926 19:27:02.195286 3467 tablet_replica.cc:330] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: stopping tablet replica
I20260926 19:27:02.195611 3468 raft_consensus.cc:2229] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:02.195616 3467 raft_consensus.cc:2229] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:02.195982 3468 raft_consensus.cc:2258] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:02.195982 3467 raft_consensus.cc:2258] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:02.197992 3467 ts_tablet_manager.cc:1905] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:02.198396 3468 ts_tablet_manager.cc:1905] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:02.201251 3423 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Bootstrap starting.
I20260926 19:27:02.201874 3334 tablet_service.cc:1467] Processing CreateTablet for tablet 41f853e5dd6f4edca374acd67312ed9f (DEFAULT_TABLE table=test-table [id=1f0f8ca2fc0746ee8706c368393bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:02.202296 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f853e5dd6f4edca374acd67312ed9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:02.202927 3423 tablet_bootstrap.cc:654] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:02.197080 3055 tablet_service.cc:1467] Processing CreateTablet for tablet 41f853e5dd6f4edca374acd67312ed9f (DEFAULT_TABLE table=test-table [id=1f0f8ca2fc0746ee8706c368393bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:02.206630 3055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f853e5dd6f4edca374acd67312ed9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:02.206919 3467 ts_tablet_manager.cc:1918] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:02.207144 3467 log.cc:1196] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/4b1315c0c4694f808e166447186d50ee
I20260926 19:27:02.207747 3467 ts_tablet_manager.cc:1939] T 4b1315c0c4694f808e166447186d50ee P 209bef2e40244773b02b8eff4e4e5505: Deleting consensus metadata
I20260926 19:27:02.209616 2904 catalog_manager.cc:4841] TS 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277): tablet 4b1315c0c4694f808e166447186d50ee (table test-table [id=805dc46fe49a4057854300f381418b51]) successfully deleted
I20260926 19:27:02.212687 3426 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Bootstrap starting.
I20260926 19:27:02.213727 3424 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Bootstrap starting.
I20260926 19:27:02.213850 3470 tablet_replica.cc:330] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: stopping tablet replica
I20260926 19:27:02.214291 3470 raft_consensus.cc:2229] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:02.214684 3426 tablet_bootstrap.cc:654] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:02.214915 3470 raft_consensus.cc:2258] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:02.215243 3424 tablet_bootstrap.cc:654] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:02.223101 3468 ts_tablet_manager.cc:1918] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:02.223302 3468 log.cc:1196] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/4b1315c0c4694f808e166447186d50ee
I20260926 19:27:02.223855 3468 ts_tablet_manager.cc:1939] T 4b1315c0c4694f808e166447186d50ee P 5aa4f831cd994adb97442cfd3de6610b: Deleting consensus metadata
I20260926 19:27:02.225800 3424 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: No bootstrap required, opened a new log
I20260926 19:27:02.226079 3424 ts_tablet_manager.cc:1397] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.010s
I20260926 19:27:02.226738 3470 ts_tablet_manager.cc:1905] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:02.226836 3424 raft_consensus.cc:348] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.227140 3424 raft_consensus.cc:374] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:02.227228 3424 raft_consensus.cc:729] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Initialized, Role: FOLLOWER
I20260926 19:27:02.227501 3424 consensus_queue.cc:260] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.230528 3424 ts_tablet_manager.cc:1428] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:27:02.231314 2904 catalog_manager.cc:4841] TS 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435): tablet 4b1315c0c4694f808e166447186d50ee (table test-table [id=805dc46fe49a4057854300f381418b51]) successfully deleted
I20260926 19:27:02.232766 3423 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: No bootstrap required, opened a new log
I20260926 19:27:02.232935 3423 ts_tablet_manager.cc:1397] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.013s
I20260926 19:27:02.233791 3423 raft_consensus.cc:348] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.234046 3426 tablet_bootstrap.cc:492] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: No bootstrap required, opened a new log
I20260926 19:27:02.234047 3423 raft_consensus.cc:374] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:02.234125 3423 raft_consensus.cc:729] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Initialized, Role: FOLLOWER
I20260926 19:27:02.234233 3426 ts_tablet_manager.cc:1397] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.015s
I20260926 19:27:02.234261 3470 ts_tablet_manager.cc:1918] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:02.234457 3470 log.cc:1196] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/4b1315c0c4694f808e166447186d50ee
I20260926 19:27:02.234367 3423 consensus_queue.cc:260] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.234987 3423 ts_tablet_manager.cc:1428] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 19:27:02.235024 3470 ts_tablet_manager.cc:1939] T 4b1315c0c4694f808e166447186d50ee P 4e63721b603245bebcd2f029db93f9c8: Deleting consensus metadata
I20260926 19:27:02.235667 3426 raft_consensus.cc:348] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.235888 3426 raft_consensus.cc:374] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:02.235975 3426 raft_consensus.cc:729] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Initialized, Role: FOLLOWER
I20260926 19:27:02.236203 3426 consensus_queue.cc:260] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.236917 3426 ts_tablet_manager.cc:1428] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 19:27:02.237213 2904 catalog_manager.cc:4841] TS 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729): tablet 4b1315c0c4694f808e166447186d50ee (table test-table [id=805dc46fe49a4057854300f381418b51]) successfully deleted
I20260926 19:27:02.273847 3430 raft_consensus.cc:482] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:02.274142 3430 raft_consensus.cc:504] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.276049 3430 leader_election.cc:288] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:02.286072 3434 raft_consensus.cc:482] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:02.286274 3434 raft_consensus.cc:504] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.286716 3421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.276424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:55905 (local address 127.1.39.1:40729)
0926 19:27:02.276632 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:02.276638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:02.276958 (+ 320us) server_negotiation.cc:407] Connection header received
0926 19:27:02.277278 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:02.277309 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:02.277415 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:02.277496 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:02.278427 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.279880 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.283262 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.285086 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.285243 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.285424 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:02.286106 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 19:27:02.286276 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:27:02.287067 3434 leader_election.cc:288] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:02.287559 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b" is_pre_election: true
I20260926 19:27:02.287869 3220 raft_consensus.cc:2454] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:02.288538 3007 leader_election.cc:302] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e63721b603245bebcd2f029db93f9c8, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:02.288831 3473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.276286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.1.39.2:55905)
0926 19:27:02.276847 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:02.276866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:02.277067 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:02.277735 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:02.277742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:02.277777 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:02.278198 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.278212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.280119 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.280127 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.283031 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.283051 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.285591 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.285600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.285855 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.285909 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 19:27:02.287258 (+ 1349us) client_negotiation.cc:240] Negotiation successful
0926 19:27:02.287396 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":253,"threads_started":1}
I20260926 19:27:02.288820 3434 raft_consensus.cc:2781] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:02.288933 3434 raft_consensus.cc:482] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:02.289026 3434 raft_consensus.cc:3037] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:02.289208 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505" is_pre_election: true
I20260926 19:27:02.289497 3359 raft_consensus.cc:2454] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 0.
I20260926 19:27:02.290792 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e63721b603245bebcd2f029db93f9c8" is_pre_election: true
I20260926 19:27:02.292291 3434 raft_consensus.cc:504] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.292737 3076 raft_consensus.cc:2379] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5aa4f831cd994adb97442cfd3de6610b in current term 1: Already voted for candidate 4e63721b603245bebcd2f029db93f9c8 in this term.
I20260926 19:27:02.293699 3434 leader_election.cc:288] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Requested vote from peers 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:02.294257 3220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b"
I20260926 19:27:02.294486 3220 raft_consensus.cc:3037] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:02.296034 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.276888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:41277 (local address 127.1.39.2:54875)
0926 19:27:02.288138 (+ 11250us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:02.288157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:02.288271 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:02.288859 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:02.288864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:02.288889 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:02.289422 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.289434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.291254 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.291259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.293421 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:02.293436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.295194 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.295202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:02.295406 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.295441 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:27:02.295533 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:27:02.295673 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11124,"thread_start_us":10875,"threads_started":1}
I20260926 19:27:02.296084 3475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:02.277020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:54875 (local address 127.1.39.3:41277)
0926 19:27:02.288409 (+ 11389us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:02.288416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:02.288437 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:02.288503 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:02.288507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:02.288580 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:02.288680 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:02.289616 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.291096 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.293603 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:02.294944 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:02.295085 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:02.295185 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:02.295614 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 19:27:02.295758 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11236,"thread_start_us":11347,"threads_started":1}
I20260926 19:27:02.296736 3359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "4e63721b603245bebcd2f029db93f9c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505"
I20260926 19:27:02.296969 3359 raft_consensus.cc:3037] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:02.296873 3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41f853e5dd6f4edca374acd67312ed9f" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505" is_pre_election: true
I20260926 19:27:02.297391 3220 raft_consensus.cc:2454] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:02.297781 3007 leader_election.cc:302] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e63721b603245bebcd2f029db93f9c8, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:02.297704 3147 leader_election.cc:302] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5aa4f831cd994adb97442cfd3de6610b; no voters: 209bef2e40244773b02b8eff4e4e5505, 4e63721b603245bebcd2f029db93f9c8
I20260926 19:27:02.298086 3434 raft_consensus.cc:2781] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:02.298173 3430 raft_consensus.cc:2726] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:27:02.298234 3434 raft_consensus.cc:686] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Running, Role: LEADER
I20260926 19:27:02.298517 3434 consensus_queue.cc:237] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:02.299974 3359 raft_consensus.cc:2454] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e63721b603245bebcd2f029db93f9c8 in term 1.
I20260926 19:27:02.301400 2917 catalog_manager.cc:5495] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "4e63721b603245bebcd2f029db93f9c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:02.334463 3220 raft_consensus.cc:1261] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Refusing update from remote peer 4e63721b603245bebcd2f029db93f9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:02.334474 3359 raft_consensus.cc:1261] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Refusing update from remote peer 4e63721b603245bebcd2f029db93f9c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:02.335100 3431 consensus_queue.cc:1035] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:02.335469 3434 consensus_queue.cc:1035] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:27:02.362845 3449 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 19:27:03.940203 2918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:03.940457 2918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:03.944391 2918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Sending DeleteTablet for 3 replicas of tablet 41f853e5dd6f4edca374acd67312ed9f
I20260926 19:27:03.945189 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f853e5dd6f4edca374acd67312ed9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260926 19:27:03.945345 3334 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f853e5dd6f4edca374acd67312ed9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260926 19:27:03.945339 3194 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f853e5dd6f4edca374acd67312ed9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45428
I20260926 19:27:03.958036 3500 tablet_replica.cc:330] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: stopping tablet replica
I20260926 19:27:03.958562 3501 tablet_replica.cc:330] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: stopping tablet replica
I20260926 19:27:03.958762 3499 tablet_replica.cc:330] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: stopping tablet replica
I20260926 19:27:03.958768 3501 raft_consensus.cc:2229] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:03.958945 3499 raft_consensus.cc:2229] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:03.959018 3501 raft_consensus.cc:2258] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:03.958967 3500 raft_consensus.cc:2229] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:03.959306 3499 raft_consensus.cc:2258] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:03.959457 3500 raft_consensus.cc:2258] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:03.960114 3501 ts_tablet_manager.cc:1905] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:03.960130 3499 ts_tablet_manager.cc:1905] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:03.960680 3500 ts_tablet_manager.cc:1905] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:03.967092 3499 ts_tablet_manager.cc:1918] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:03.967108 3501 ts_tablet_manager.cc:1918] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:03.967228 3501 log.cc:1196] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/41f853e5dd6f4edca374acd67312ed9f
I20260926 19:27:03.967295 3499 log.cc:1196] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/41f853e5dd6f4edca374acd67312ed9f
I20260926 19:27:03.967720 3501 ts_tablet_manager.cc:1939] T 41f853e5dd6f4edca374acd67312ed9f P 209bef2e40244773b02b8eff4e4e5505: Deleting consensus metadata
I20260926 19:27:03.967749 3499 ts_tablet_manager.cc:1939] T 41f853e5dd6f4edca374acd67312ed9f P 4e63721b603245bebcd2f029db93f9c8: Deleting consensus metadata
I20260926 19:27:03.967935 3500 ts_tablet_manager.cc:1918] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:03.968065 3500 log.cc:1196] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/41f853e5dd6f4edca374acd67312ed9f
I20260926 19:27:03.968485 3500 ts_tablet_manager.cc:1939] T 41f853e5dd6f4edca374acd67312ed9f P 5aa4f831cd994adb97442cfd3de6610b: Deleting consensus metadata
I20260926 19:27:03.968757 2904 catalog_manager.cc:4841] TS 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729): tablet 41f853e5dd6f4edca374acd67312ed9f (table test-table [id=1f0f8ca2fc0746ee8706c368393bc603]) successfully deleted
I20260926 19:27:03.969056 2904 catalog_manager.cc:4841] TS 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277): tablet 41f853e5dd6f4edca374acd67312ed9f (table test-table [id=1f0f8ca2fc0746ee8706c368393bc603]) successfully deleted
I20260926 19:27:03.970541 2904 catalog_manager.cc:4841] TS 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435): tablet 41f853e5dd6f4edca374acd67312ed9f (table test-table [id=1f0f8ca2fc0746ee8706c368393bc603]) successfully deleted
Sep 26 19:27:03 dist-test-slave-9wb0 krb5kdc[2874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:27:04 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:04.012099 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:03.989942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.1.39.62:42069)
0926 19:27:03.990519 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:03.990526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:03.990647 (+ 121us) server_negotiation.cc:407] Connection header received
0926 19:27:03.990856 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:03.990868 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:03.990930 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:03.991047 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:03.992293 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:03.993069 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:03.994451 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:03.994872 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:03.995015 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.006761 (+ 11746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:04.006767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:04.006770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:04.006811 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:04.009517 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:04.010202 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:04.010209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:04.010212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:04.010345 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:04.010741 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:04.010747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:04.010750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:04.011020 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:04.011280 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.011656 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.011825 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":269,"threads_started":1}
I20260926 19:27:04.015885 2918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:04.016556 2918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:04.024438 3334 tablet_service.cc:1467] Processing CreateTablet for tablet d2495b99679f4f2f99e0c43a05e4127c (DEFAULT_TABLE table=test-table [id=038c911bb67049379a06e9661f026509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.024452 3056 tablet_service.cc:1467] Processing CreateTablet for tablet d2495b99679f4f2f99e0c43a05e4127c (DEFAULT_TABLE table=test-table [id=038c911bb67049379a06e9661f026509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.024516 3194 tablet_service.cc:1467] Processing CreateTablet for tablet d2495b99679f4f2f99e0c43a05e4127c (DEFAULT_TABLE table=test-table [id=038c911bb67049379a06e9661f026509]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.024920 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2495b99679f4f2f99e0c43a05e4127c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.024927 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2495b99679f4f2f99e0c43a05e4127c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.024950 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2495b99679f4f2f99e0c43a05e4127c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.030103 3510 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Bootstrap starting.
I20260926 19:27:04.032302 3510 tablet_bootstrap.cc:654] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.036208 3510 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: No bootstrap required, opened a new log
I20260926 19:27:04.036371 3510 ts_tablet_manager.cc:1397] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 19:27:04.037084 3510 raft_consensus.cc:348] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.037292 3510 raft_consensus.cc:374] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.037432 3510 raft_consensus.cc:729] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.037776 3510 consensus_queue.cc:260] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.041867 3511 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Bootstrap starting.
I20260926 19:27:04.042456 3512 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Bootstrap starting.
I20260926 19:27:04.043243 3510 ts_tablet_manager.cc:1428] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20260926 19:27:04.044785 3511 tablet_bootstrap.cc:654] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.044878 3512 tablet_bootstrap.cc:654] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.050148 3511 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: No bootstrap required, opened a new log
I20260926 19:27:04.050246 3512 tablet_bootstrap.cc:492] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: No bootstrap required, opened a new log
I20260926 19:27:04.050329 3511 ts_tablet_manager.cc:1397] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 19:27:04.050385 3512 ts_tablet_manager.cc:1397] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260926 19:27:04.051061 3511 raft_consensus.cc:348] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.051286 3511 raft_consensus.cc:374] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.051157 3512 raft_consensus.cc:348] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.051376 3512 raft_consensus.cc:374] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.051385 3511 raft_consensus.cc:729] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.051465 3512 raft_consensus.cc:729] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.051666 3511 consensus_queue.cc:260] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.051679 3512 consensus_queue.cc:260] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.052197 3511 ts_tablet_manager.cc:1428] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 19:27:04.057854 3512 ts_tablet_manager.cc:1428] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260926 19:27:04.260461 3517 raft_consensus.cc:482] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:04.260696 3517 raft_consensus.cc:504] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.261495 3517 leader_election.cc:288] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:04.262108 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2495b99679f4f2f99e0c43a05e4127c" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e63721b603245bebcd2f029db93f9c8" is_pre_election: true
I20260926 19:27:04.262131 3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2495b99679f4f2f99e0c43a05e4127c" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505" is_pre_election: true
I20260926 19:27:04.262351 3076 raft_consensus.cc:2454] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa4f831cd994adb97442cfd3de6610b in term 0.
I20260926 19:27:04.262392 3358 raft_consensus.cc:2454] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5aa4f831cd994adb97442cfd3de6610b in term 0.
I20260926 19:27:04.263262 3147 leader_election.cc:302] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:04.263561 3517 raft_consensus.cc:2781] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:04.263715 3517 raft_consensus.cc:482] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:04.263816 3517 raft_consensus.cc:3037] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.266075 3517 raft_consensus.cc:504] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.266700 3517 leader_election.cc:288] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 election: Requested vote from peers 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729), 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277)
I20260926 19:27:04.267153 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2495b99679f4f2f99e0c43a05e4127c" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e63721b603245bebcd2f029db93f9c8"
I20260926 19:27:04.267236 3358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2495b99679f4f2f99e0c43a05e4127c" candidate_uuid: "5aa4f831cd994adb97442cfd3de6610b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209bef2e40244773b02b8eff4e4e5505"
I20260926 19:27:04.267381 3076 raft_consensus.cc:3037] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.267468 3358 raft_consensus.cc:3037] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.269477 3358 raft_consensus.cc:2454] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa4f831cd994adb97442cfd3de6610b in term 1.
I20260926 19:27:04.269582 3076 raft_consensus.cc:2454] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aa4f831cd994adb97442cfd3de6610b in term 1.
I20260926 19:27:04.269930 3147 leader_election.cc:302] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:04.270229 3517 raft_consensus.cc:2781] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:04.270653 3517 raft_consensus.cc:686] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 1 LEADER]: Becoming Leader. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Running, Role: LEADER
I20260926 19:27:04.271062 3517 consensus_queue.cc:237] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.274243 2917 catalog_manager.cc:5495] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2). New cstate: current_term: 1 leader_uuid: "5aa4f831cd994adb97442cfd3de6610b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:04.343128 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.337037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.1.39.2:35435)
0926 19:27:04.338036 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.338040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.338058 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:27:04.338136 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.338140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.338211 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.338347 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.338791 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.339615 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.340919 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.341469 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.341575 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.341970 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.342580 (+ 610us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.342704 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.342799 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.342863 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":791,"threads_started":1}
I20260926 19:27:04.358934 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.352889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:50067 (local address 127.1.39.1:40729)
0926 19:27:04.353459 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.353463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.353848 (+ 385us) server_negotiation.cc:407] Connection header received
0926 19:27:04.353997 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.354001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.354066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.354146 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.354694 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.355451 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.356862 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.357276 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.357431 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.358271 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.358382 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.358451 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.358616 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.358674 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":307,"threads_started":1}
I20260926 19:27:04.359004 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.352887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.1.39.2:50067)
0926 19:27:04.353763 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.353779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.353895 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.354187 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.354190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.354198 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:04.354542 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.354550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.355568 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.355571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.356744 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.356755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.357502 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.357506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.358034 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.358098 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.358498 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.358509 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.358586 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.358698 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":585,"threads_started":1}
I20260926 19:27:04.367156 3358 raft_consensus.cc:1261] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Refusing update from remote peer 5aa4f831cd994adb97442cfd3de6610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:04.367393 3076 raft_consensus.cc:1261] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Refusing update from remote peer 5aa4f831cd994adb97442cfd3de6610b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:04.367815 3518 consensus_queue.cc:1035] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:04.368201 3517 consensus_queue.cc:1035] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:04.374006 3480 mvcc.cc:205] Tried to move back new op lower bound from 7333686576601923584 to 7333686576217669632. Current Snapshot: MvccSnapshot[applied={T|T < 7333686576601923584}]
I20260926 19:27:04.402276 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.397006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:53059 (local address 127.1.39.2:35435)
0926 19:27:04.397170 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.397178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.397288 (+ 110us) server_negotiation.cc:407] Connection header received
0926 19:27:04.397475 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.397479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.397538 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.397616 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.397971 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.398800 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.400622 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.401032 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.401170 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.401586 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.401692 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.401790 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.401900 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.401971 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:27:04.402737 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.396653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35435 (local address 127.1.39.1:53059)
0926 19:27:04.397198 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.397219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.397373 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.397622 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.397624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.397633 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:04.397851 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.397858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.398960 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.398964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.400454 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.400468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.401200 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.401207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.401382 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.401418 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.401792 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.401806 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.401883 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.402003 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":261,"threads_started":1}
I20260926 19:27:04.433707 2917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:04.433969 2917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:04.437201 2917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Sending DeleteTablet for 3 replicas of tablet d2495b99679f4f2f99e0c43a05e4127c
I20260926 19:27:04.438117 3194 tablet_service.cc:1514] Processing DeleteTablet for tablet d2495b99679f4f2f99e0c43a05e4127c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:04 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45428
I20260926 19:27:04.438083 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet d2495b99679f4f2f99e0c43a05e4127c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:04 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260926 19:27:04.438269 3334 tablet_service.cc:1514] Processing DeleteTablet for tablet d2495b99679f4f2f99e0c43a05e4127c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:04 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260926 19:27:04.438411 3500 tablet_replica.cc:330] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: stopping tablet replica
I20260926 19:27:04.438411 3499 tablet_replica.cc:330] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: stopping tablet replica
I20260926 19:27:04.438602 3499 raft_consensus.cc:2229] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:04.438611 3500 raft_consensus.cc:2229] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:04.438788 3499 raft_consensus.cc:2258] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:04.439062 3501 tablet_replica.cc:330] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: stopping tablet replica
I20260926 19:27:04.439057 3500 raft_consensus.cc:2258] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:04.439256 3501 raft_consensus.cc:2229] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:04.439440 3501 raft_consensus.cc:2258] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:04.439685 2917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:04.440346 2917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:04.440912 3500 ts_tablet_manager.cc:1905] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:04.441229 3499 ts_tablet_manager.cc:1905] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:04.447773 3500 ts_tablet_manager.cc:1918] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:04.447803 3499 ts_tablet_manager.cc:1918] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:04.448006 3499 log.cc:1196] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/d2495b99679f4f2f99e0c43a05e4127c
I20260926 19:27:04.448052 3334 tablet_service.cc:1467] Processing CreateTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (DEFAULT_TABLE table=test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.448462 3499 ts_tablet_manager.cc:1939] T d2495b99679f4f2f99e0c43a05e4127c P 4e63721b603245bebcd2f029db93f9c8: Deleting consensus metadata
I20260926 19:27:04.448488 3334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.448632 3194 tablet_service.cc:1467] Processing CreateTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (DEFAULT_TABLE table=test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.449105 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.449473 2904 catalog_manager.cc:4841] TS 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729): tablet d2495b99679f4f2f99e0c43a05e4127c (table test-table [id=038c911bb67049379a06e9661f026509]) successfully deleted
I20260926 19:27:04.450629 3501 ts_tablet_manager.cc:1905] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:04.455847 3512 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Bootstrap starting.
I20260926 19:27:04.456403 3500 log.cc:1196] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/d2495b99679f4f2f99e0c43a05e4127c
I20260926 19:27:04.457028 3500 ts_tablet_manager.cc:1939] T d2495b99679f4f2f99e0c43a05e4127c P 5aa4f831cd994adb97442cfd3de6610b: Deleting consensus metadata
I20260926 19:27:04.457647 3501 ts_tablet_manager.cc:1918] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:04.457800 3501 log.cc:1196] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/d2495b99679f4f2f99e0c43a05e4127c
I20260926 19:27:04.457923 3512 tablet_bootstrap.cc:654] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.458227 3501 ts_tablet_manager.cc:1939] T d2495b99679f4f2f99e0c43a05e4127c P 209bef2e40244773b02b8eff4e4e5505: Deleting consensus metadata
I20260926 19:27:04.458354 2904 catalog_manager.cc:4841] TS 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435): tablet d2495b99679f4f2f99e0c43a05e4127c (table test-table [id=038c911bb67049379a06e9661f026509]) successfully deleted
I20260926 19:27:04.459264 2904 catalog_manager.cc:4841] TS 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277): tablet d2495b99679f4f2f99e0c43a05e4127c (table test-table [id=038c911bb67049379a06e9661f026509]) successfully deleted
I20260926 19:27:04.459702 3056 tablet_service.cc:1467] Processing CreateTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (DEFAULT_TABLE table=test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:04.460090 3056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:04.461913 3510 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Bootstrap starting.
I20260926 19:27:04.462767 3512 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: No bootstrap required, opened a new log
I20260926 19:27:04.462970 3512 ts_tablet_manager.cc:1397] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 19:27:04.463560 3510 tablet_bootstrap.cc:654] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.463781 3512 raft_consensus.cc:348] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.464039 3512 raft_consensus.cc:374] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.464129 3512 raft_consensus.cc:729] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aa4f831cd994adb97442cfd3de6610b, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.464370 3512 consensus_queue.cc:260] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.465750 3512 ts_tablet_manager.cc:1428] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 19:27:04.465791 3510 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: No bootstrap required, opened a new log
I20260926 19:27:04.466027 3510 ts_tablet_manager.cc:1397] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260926 19:27:04.466744 3510 raft_consensus.cc:348] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.466981 3510 raft_consensus.cc:374] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.467059 3510 raft_consensus.cc:729] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.467288 3510 consensus_queue.cc:260] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.468276 3510 ts_tablet_manager.cc:1428] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:27:04.471242 3511 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Bootstrap starting.
I20260926 19:27:04.473253 3511 tablet_bootstrap.cc:654] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:04.475216 3511 tablet_bootstrap.cc:492] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: No bootstrap required, opened a new log
I20260926 19:27:04.475370 3511 ts_tablet_manager.cc:1397] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 19:27:04.476064 3511 raft_consensus.cc:348] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.476289 3511 raft_consensus.cc:374] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:04.476370 3511 raft_consensus.cc:729] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e63721b603245bebcd2f029db93f9c8, State: Initialized, Role: FOLLOWER
I20260926 19:27:04.476599 3511 consensus_queue.cc:260] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.477236 3511 ts_tablet_manager.cc:1428] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:27:04.575856 3514 raft_consensus.cc:482] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:04.576074 3514 raft_consensus.cc:504] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.577706 3514 leader_election.cc:288] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729), 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435)
I20260926 19:27:04.585417 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.578021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:54935 (local address 127.1.39.1:40729)
0926 19:27:04.578223 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.578230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.578649 (+ 419us) server_negotiation.cc:407] Connection header received
0926 19:27:04.578981 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.578990 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.579085 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.579160 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:04.580108 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.581371 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.583014 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.584450 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.584587 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.584702 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.584983 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.585129 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:27:04.585454 3536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.577885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.1.39.3:54935)
0926 19:27:04.578542 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.578561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.578780 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.579410 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.579419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.579451 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:04.579882 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.579896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.581526 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.581534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.582856 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.582869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.584650 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.584656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.584815 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.584849 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.584944 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.585073 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
I20260926 19:27:04.586560 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d1d61c1cc44b09806eb36fb3b6fdfa" candidate_uuid: "209bef2e40244773b02b8eff4e4e5505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e63721b603245bebcd2f029db93f9c8" is_pre_election: true
I20260926 19:27:04.586712 3537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.578589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35435 (local address 127.1.39.3:36957)
0926 19:27:04.580466 (+ 1877us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.580483 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.580678 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.581003 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.581008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.581032 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:04.581390 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.581407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.582876 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.582882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.584267 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.584283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.585928 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.585935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.586141 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.586175 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.586273 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.586402 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"thread_start_us":1619,"threads_started":1}
I20260926 19:27:04.586817 3076 raft_consensus.cc:2454] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209bef2e40244773b02b8eff4e4e5505 in term 0.
I20260926 19:27:04.587249 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.578817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:36957 (local address 127.1.39.2:35435)
0926 19:27:04.579025 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.579031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.580575 (+ 1544us) server_negotiation.cc:407] Connection header received
0926 19:27:04.580829 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.580833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.580893 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.580998 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:04.581616 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.582729 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.584409 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.585700 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.585855 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.585946 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.586312 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.586464 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:27:04.587466 3286 leader_election.cc:302] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 4e63721b603245bebcd2f029db93f9c8; no voters:
I20260926 19:27:04.587914 3514 raft_consensus.cc:2781] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:04.588042 3514 raft_consensus.cc:482] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:04.588136 3514 raft_consensus.cc:3037] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.588459 3219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d1d61c1cc44b09806eb36fb3b6fdfa" candidate_uuid: "209bef2e40244773b02b8eff4e4e5505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b" is_pre_election: true
I20260926 19:27:04.588726 3219 raft_consensus.cc:2454] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 209bef2e40244773b02b8eff4e4e5505 in term 0.
I20260926 19:27:04.591014 3514 raft_consensus.cc:504] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.591799 3514 leader_election.cc:288] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [CANDIDATE]: Term 1 election: Requested vote from peers 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729), 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435)
I20260926 19:27:04.592233 3076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d1d61c1cc44b09806eb36fb3b6fdfa" candidate_uuid: "209bef2e40244773b02b8eff4e4e5505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e63721b603245bebcd2f029db93f9c8"
I20260926 19:27:04.592435 3076 raft_consensus.cc:3037] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.592571 3219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d1d61c1cc44b09806eb36fb3b6fdfa" candidate_uuid: "209bef2e40244773b02b8eff4e4e5505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aa4f831cd994adb97442cfd3de6610b"
I20260926 19:27:04.592783 3219 raft_consensus.cc:3037] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:04.595427 3076 raft_consensus.cc:2454] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209bef2e40244773b02b8eff4e4e5505 in term 1.
I20260926 19:27:04.596143 3219 raft_consensus.cc:2454] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 209bef2e40244773b02b8eff4e4e5505 in term 1.
I20260926 19:27:04.596576 3286 leader_election.cc:302] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 209bef2e40244773b02b8eff4e4e5505, 5aa4f831cd994adb97442cfd3de6610b; no voters:
I20260926 19:27:04.596993 3514 raft_consensus.cc:2781] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:04.597517 3514 raft_consensus.cc:686] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 1 LEADER]: Becoming Leader. State: Replica: 209bef2e40244773b02b8eff4e4e5505, State: Running, Role: LEADER
I20260926 19:27:04.598075 3514 consensus_queue.cc:237] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } }
I20260926 19:27:04.601574 2919 catalog_manager.cc:5495] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "209bef2e40244773b02b8eff4e4e5505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "209bef2e40244773b02b8eff4e4e5505" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 41277 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:04.642378 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.635913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.1.39.3:41277)
0926 19:27:04.636538 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.636542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.636560 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:27:04.636641 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.636644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.636716 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.636914 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.637508 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.638382 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.640081 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.640503 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.640633 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.641092 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.641730 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.641895 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.641965 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.642045 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":362,"threads_started":1}
I20260926 19:27:04.659710 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.653416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40729 (local address 127.1.39.3:42519)
0926 19:27:04.654138 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.654155 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.654279 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.654671 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.654675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.654684 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:04.654934 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.654941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.656353 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.656357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.657683 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.657694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.658671 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.658675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.658788 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.658839 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.659246 (+ 407us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.659253 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.659313 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.659408 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":448,"threads_started":1}
I20260926 19:27:04.659778 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.653652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:42519 (local address 127.1.39.1:40729)
0926 19:27:04.653841 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.653848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.654248 (+ 400us) server_negotiation.cc:407] Connection header received
0926 19:27:04.654422 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.654428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.654523 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.654626 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.655079 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.656199 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.657828 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.658491 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.658631 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.659016 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.659148 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.659241 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.659382 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.659474 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:27:04.669216 3076 raft_consensus.cc:1261] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Refusing update from remote peer 209bef2e40244773b02b8eff4e4e5505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:04.669454 3219 raft_consensus.cc:1261] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Refusing update from remote peer 209bef2e40244773b02b8eff4e4e5505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:04.670012 3538 consensus_queue.cc:1035] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e63721b603245bebcd2f029db93f9c8" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:04.670465 3514 consensus_queue.cc:1035] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aa4f831cd994adb97442cfd3de6610b" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:04.675433 3483 mvcc.cc:205] Tried to move back new op lower bound from 7333686577838403584 to 7333686577557254144. Current Snapshot: MvccSnapshot[applied={T|T < 7333686577838403584}]
I20260926 19:27:04.706153 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.700167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:41277 (local address 127.1.39.1:57143)
0926 19:27:04.700904 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:04.700928 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:04.701066 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:04.701382 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:04.701387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:04.701399 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:04.701573 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.701582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.702697 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.702703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.704257 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:04.704270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.704946 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.704952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:04.705136 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.705185 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.705627 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.705643 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 19:27:04.705741 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:27:04.705871 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599}
I20260926 19:27:04.706257 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:04.700701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57143 (local address 127.1.39.3:41277)
0926 19:27:04.700849 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:04.700854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:04.700999 (+ 145us) server_negotiation.cc:407] Connection header received
0926 19:27:04.701175 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:04.701179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:04.701236 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:04.701366 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:04.701700 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.702541 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.704391 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:04.704796 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:04.704922 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:04.705386 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:04.705516 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:04.705615 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:04.705765 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 19:27:04.705840 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 19:27:04.708195 3483 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 19:27:06.298071 2919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:06.298300 2919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:06.302065 2919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7fc4b154637a42b9afd124bb2872dc63: Sending DeleteTablet for 3 replicas of tablet f1d1d61c1cc44b09806eb36fb3b6fdfa
I20260926 19:27:06.302894 3056 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59198
I20260926 19:27:06.302902 3194 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45428
I20260926 19:27:06.303002 3334 tablet_service.cc:1514] Processing DeleteTablet for tablet f1d1d61c1cc44b09806eb36fb3b6fdfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:45274
I20260926 19:27:06.304800 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2990
I20260926 19:27:06.318583 3573 tablet_replica.cc:330] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: stopping tablet replica
I20260926 19:27:06.318929 3573 raft_consensus.cc:2229] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:06.319852 3573 raft_consensus.cc:2258] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:06.321925 3573 ts_tablet_manager.cc:1905] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:06.326958 3572 tablet_replica.cc:330] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: stopping tablet replica
I20260926 19:27:06.327198 3572 raft_consensus.cc:2229] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:06.327901 3572 raft_consensus.cc:2258] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:06.329896 3572 ts_tablet_manager.cc:1905] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:06.331852 3573 ts_tablet_manager.cc:1918] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:06.332206 3573 log.cc:1196] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/f1d1d61c1cc44b09806eb36fb3b6fdfa
I20260926 19:27:06.332953 3573 ts_tablet_manager.cc:1939] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 209bef2e40244773b02b8eff4e4e5505: Deleting consensus metadata
I20260926 19:27:06.334259 3571 tablet_replica.cc:330] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: stopping tablet replica
I20260926 19:27:06.334520 3571 raft_consensus.cc:2229] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:06.334637 2904 catalog_manager.cc:4841] TS 209bef2e40244773b02b8eff4e4e5505 (127.1.39.3:41277): tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (table test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]) successfully deleted
I20260926 19:27:06.334781 3571 raft_consensus.cc:2258] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:06.336055 3571 ts_tablet_manager.cc:1905] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:06.338101 3572 ts_tablet_manager.cc:1918] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:06.338305 3572 log.cc:1196] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/f1d1d61c1cc44b09806eb36fb3b6fdfa
I20260926 19:27:06.338826 3572 ts_tablet_manager.cc:1939] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 4e63721b603245bebcd2f029db93f9c8: Deleting consensus metadata
I20260926 19:27:06.339787 2904 catalog_manager.cc:4841] TS 4e63721b603245bebcd2f029db93f9c8 (127.1.39.1:40729): tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (table test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]) successfully deleted
I20260926 19:27:06.342913 3571 ts_tablet_manager.cc:1918] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 19:27:06.343065 3571 log.cc:1196] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/f1d1d61c1cc44b09806eb36fb3b6fdfa
I20260926 19:27:06.343559 3571 ts_tablet_manager.cc:1939] T f1d1d61c1cc44b09806eb36fb3b6fdfa P 5aa4f831cd994adb97442cfd3de6610b: Deleting consensus metadata
I20260926 19:27:06.344769 2904 catalog_manager.cc:4841] TS 5aa4f831cd994adb97442cfd3de6610b (127.1.39.2:35435): tablet f1d1d61c1cc44b09806eb36fb3b6fdfa (table test-table [id=97ac76fa6fc24c94b9a2569e2060bbeb]) successfully deleted
I20260926 19:27:06.345216 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:06.306590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.1.39.1:40729)
0926 19:27:06.325237 (+ 18647us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:06.325244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:06.326203 (+ 959us) server_negotiation.cc:407] Connection header received
0926 19:27:06.326486 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:06.326494 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:06.326609 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:06.326745 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:06.328341 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.329104 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.330984 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.331667 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.331762 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:06.337557 (+ 5795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:06.337576 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:06.337599 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:06.337645 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:06.342155 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.342850 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.342860 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.342863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.343049 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.343604 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.343610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.343613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.344117 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:06.344438 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:06.344619 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:27:06.344859 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18488,"thread_start_us":18704,"threads_started":1}
W20260926 19:27:06.346045 3121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59218
W20260926 19:27:06.346748 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:06.346971 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2990
I20260926 19:27:06.374279 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3130
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:06.389067 3576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:06.375027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.1.39.2:35435)
0926 19:27:06.375568 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:06.375572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:06.375589 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:27:06.375640 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:06.375644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:06.375705 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:06.375797 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:06.376971 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.377761 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.378946 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.379711 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.379805 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:06.383920 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:06.383934 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:06.383950 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:06.383982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:06.386749 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.387346 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.387353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.387355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.387487 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.387775 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.387778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.387780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.388107 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:06.388366 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:06.388614 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 19:27:06.388770 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":294,"threads_started":1}
W20260926 19:27:06.389881 3260 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45458
W20260926 19:27:06.390472 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:06.390626 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3130
I20260926 19:27:06.413283 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3269
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[2874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450823, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:06.429445 3577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:06.414160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.1.39.3:41277)
0926 19:27:06.414686 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:06.414691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:06.414711 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:27:06.414769 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:06.414772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:06.414837 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:06.414955 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:06.416245 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.416986 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.418285 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:06.418764 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:06.418887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:06.423467 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:06.423484 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:06.423502 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:06.423540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:06.426782 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.427522 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.427530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.427532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.427676 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:06.428030 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:06.428033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:06.428035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:06.428390 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:06.428610 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:06.428860 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 19:27:06.429019 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":283,"threads_started":1}
W20260926 19:27:06.430094 3399 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45282
W20260926 19:27:06.430708 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3269: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:06.430878 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3269
I20260926 19:27:06.452713 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 2887
I20260926 19:27:06.453902 2959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:06.570876 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 2887
2026-09-26T19:27:06Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8382 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[3581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[3581](info): set up 2 sockets
Sep 26 19:27:06 dist-test-slave-9wb0 krb5kdc[3581](info): commencing operation
krb5kdc: starting...
W20260926 19:27:07.689723 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:07 dist-test-slave-9wb0 krb5kdc[3581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:07.881095 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40587
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:08.200981 3594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:08.201347 3594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:08.201516 3594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:08.208770 3594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:08.208912 3594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:08.209020 3594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:08.209128 3594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:08.209219 3594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:08.218922 3594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:40587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:08.220278 3594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:08.221658 3594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:08.233559 3599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:08.233530 3602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:08.233623 3600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:08.234172 3594 server_base.cc:1034] running on GCE node
I20260926 19:27:08.235159 3594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:08.236680 3594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:08.237926 3594 hybrid_clock.cc:648] HybridClock initialized: now 1790450828237889 us; error 58 us; skew 500 ppm
I20260926 19:27:08.238300 3594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:08 dist-test-slave-9wb0 krb5kdc[3581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:08.241641 3594 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:08.243166 3594 webserver.cc:460] Webserver started at http://127.1.39.62:35209/ using document root <none> and password file <none>
I20260926 19:27:08.243870 3594 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:08.244016 3594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:08.244347 3594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:08.247572 3594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "95bacda59c9343908bc62b0762b0da9f"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "1ffca9d93897dffd1eb6d68b36534c55"
server_key_iv: "6f60e0cc49b9c283622893997f0db426"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.248348 3594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "95bacda59c9343908bc62b0762b0da9f"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "1ffca9d93897dffd1eb6d68b36534c55"
server_key_iv: "6f60e0cc49b9c283622893997f0db426"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.253662 3594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:27:08.257103 3609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:08.258555 3594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:08.258742 3594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "95bacda59c9343908bc62b0762b0da9f"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "1ffca9d93897dffd1eb6d68b36534c55"
server_key_iv: "6f60e0cc49b9c283622893997f0db426"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.258944 3594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:08.310735 3594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:08.315485 3594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:08.315801 3594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:08.334708 3594 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:40587
I20260926 19:27:08.334731 3660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:40587 every 8 connection(s)
I20260926 19:27:08.336143 3594 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:08.339318 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3594
I20260926 19:27:08.339536 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:08.340323 1180 external_mini_cluster.cc:1428] Setting key 35d683f312bdf5d7349cfca11c79667f
I20260926 19:27:08.340368 3661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:08.354844 3661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f: Bootstrap starting.
Sep 26 19:27:08 dist-test-slave-9wb0 krb5kdc[3581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:08.358098 3661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:08.359158 3661 log.cc:824] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:08.361975 3661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f: No bootstrap required, opened a new log
I20260926 19:27:08.363922 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:08.342389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60460 (local address 127.1.39.62:40587)
0926 19:27:08.343193 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:08.343204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:08.343247 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:27:08.345080 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:08.345102 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:08.345555 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:08.345898 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:08.347704 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.349009 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.350766 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.351488 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.351632 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:08.356334 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:08.356343 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:08.356358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:08.356394 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:08.359667 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:08.360376 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:08.360383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:08.360389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:08.360561 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:08.360939 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:08.360944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:08.360952 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:08.361522 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:08.361810 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:08.362257 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 19:27:08.362746 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":383,"threads_started":1}
I20260926 19:27:08.371150 3661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } }
I20260926 19:27:08.371589 3661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:08.371706 3661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bacda59c9343908bc62b0762b0da9f, State: Initialized, Role: FOLLOWER
I20260926 19:27:08.372300 3661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } }
I20260926 19:27:08.372557 3661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:08.372700 3661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:08.372877 3661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:08.375119 3661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } }
I20260926 19:27:08.375640 3661 leader_election.cc:302] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95bacda59c9343908bc62b0762b0da9f; no voters:
I20260926 19:27:08.376263 3661 leader_election.cc:288] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:08.376396 3666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:08.377087 3666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [term 1 LEADER]: Becoming Leader. State: Replica: 95bacda59c9343908bc62b0762b0da9f, State: Running, Role: LEADER
I20260926 19:27:08.377818 3666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } }
I20260926 19:27:08.378611 3661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:08.380787 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bacda59c9343908bc62b0762b0da9f. Latest consensus state: current_term: 1 leader_uuid: "95bacda59c9343908bc62b0762b0da9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } } }
I20260926 19:27:08.380980 3667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bacda59c9343908bc62b0762b0da9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bacda59c9343908bc62b0762b0da9f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40587 } } }
I20260926 19:27:08.381248 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:08.381316 3667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:08.383201 3675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:08.388109 3675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:08.398823 3675 catalog_manager.cc:1347] Generated new cluster ID: 50d1f454674d4c1bb35210e24158b949
I20260926 19:27:08.399000 3675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:08.410651 3675 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:08.412495 3675 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:08.429971 3675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95bacda59c9343908bc62b0762b0da9f: Generated new TSK 0
I20260926 19:27:08.431005 3675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:08.489459 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:08.832930 3687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:08.833189 3687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:08.833281 3687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:08.840335 3687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:08.840458 3687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:08.840626 3687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:08.850013 3687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:08.851289 3687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:08.852495 3687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:08.865825 3695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:08.865885 3692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:08.865798 3693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:08.866804 3687 server_base.cc:1034] running on GCE node
I20260926 19:27:08.867311 3687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:08.868165 3687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:08.869400 3687 hybrid_clock.cc:648] HybridClock initialized: now 1790450828869367 us; error 32 us; skew 500 ppm
I20260926 19:27:08.869680 3687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:08 dist-test-slave-9wb0 krb5kdc[3581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:08.872655 3687 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:08.874109 3687 webserver.cc:460] Webserver started at http://127.1.39.1:36417/ using document root <none> and password file <none>
I20260926 19:27:08.874778 3687 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:08.874907 3687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:08.875229 3687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:08.878078 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "82d34962f46f42abbda395e2f54c96c9"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "2817561cc506bf0059f0626df55b3ecb"
server_key_iv: "18f2f7306820ed6b693c9916b4a31dd8"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.878898 3687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "82d34962f46f42abbda395e2f54c96c9"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "2817561cc506bf0059f0626df55b3ecb"
server_key_iv: "18f2f7306820ed6b693c9916b4a31dd8"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.884238 3687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:27:08.887677 3702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:08.889005 3687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:27:08.889191 3687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "82d34962f46f42abbda395e2f54c96c9"
format_stamp: "Formatted at 2026-09-26 19:27:08 on dist-test-slave-9wb0"
server_key: "2817561cc506bf0059f0626df55b3ecb"
server_key_iv: "18f2f7306820ed6b693c9916b4a31dd8"
server_key_version: "encryptionkey@0"
I20260926 19:27:08.889441 3687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:08.901940 3687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:08.906164 3687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:08.906474 3687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:08.907444 3687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:08.908772 3687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:08.908874 3687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:08.909024 3687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:08.909096 3687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:08.942895 3687 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33143
I20260926 19:27:08.942941 3814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33143 every 8 connection(s)
I20260926 19:27:08.944154 3687 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:08.952735 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3687
I20260926 19:27:08.952968 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:08.953649 1180 external_mini_cluster.cc:1428] Setting key 023d7c36ef2c952a73da4847df7114e1
Sep 26 19:27:08 dist-test-slave-9wb0 krb5kdc[3581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450828, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:08.964985 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:08.946729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57261 (local address 127.1.39.62:40587)
0926 19:27:08.947144 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:08.947149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:08.948148 (+ 999us) server_negotiation.cc:407] Connection header received
0926 19:27:08.949926 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:08.949930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:08.950007 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:08.950103 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:08.951549 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.952285 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.954336 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.954875 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.954984 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:08.958541 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:08.958550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:08.958557 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:08.958599 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:08.961768 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:08.962706 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:08.962710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:08.962712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:08.962841 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:08.963311 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:08.963314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:08.963316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:08.963554 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:08.963708 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:08.964572 (+ 864us) server_negotiation.cc:299] Negotiation successful
0926 19:27:08.964729 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":219,"threads_started":1}
I20260926 19:27:08.966524 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:08.947316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40587 (local address 127.1.39.1:57261)
0926 19:27:08.947998 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:08.948036 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:08.949677 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:08.950318 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:08.950325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:08.950836 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:08.951372 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:08.951388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.952552 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.952562 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:08.954162 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:08.954189 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:08.955129 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:08.955136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:08.955451 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:08.956177 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:08.956199 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:08.958318 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:08.961898 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:08.961907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:08.961926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:08.962579 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:08.962936 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:08.962944 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:08.962951 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:08.963189 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:08.963709 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:08.963720 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:08.964176 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 19:27:08.964494 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 19:27:08.965057 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260926 19:27:08.968060 3815 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40587
I20260926 19:27:08.968362 3815 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:08.969111 3815 heartbeater.cc:511] Master 127.1.39.62:40587 requested a full tablet report, sending...
I20260926 19:27:08.971272 3626 ts_manager.cc:194] Registered new tserver with Master: 82d34962f46f42abbda395e2f54c96c9 (127.1.39.1:33143)
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:27:08.972682 3626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:57261
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:08.997076 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:09.314647 3821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:09.314965 3821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:09.315132 3821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:09.322423 3821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:09.322570 3821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:09.322746 3821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:09.332347 3821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:09.333683 3821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:09.335093 3821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:09.348153 3829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:09.348176 3827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:09.348929 3826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:09.349123 3821 server_base.cc:1034] running on GCE node
I20260926 19:27:09.349803 3821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:09.350883 3821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:09.352159 3821 hybrid_clock.cc:648] HybridClock initialized: now 1790450829352101 us; error 46 us; skew 500 ppm
I20260926 19:27:09.352460 3821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:09 dist-test-slave-9wb0 krb5kdc[3581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:09.360812 3821 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:09.362283 3821 webserver.cc:460] Webserver started at http://127.1.39.2:45091/ using document root <none> and password file <none>
I20260926 19:27:09.363096 3821 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:09.363238 3821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:09.363575 3821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:09.366443 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "cfab11ee08c84aed9f041d96cd10720f"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "6f3d85e419af16990f09df9c2d13531d"
server_key_iv: "fca682f6a71f1ad8994104db76cdc063"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.367224 3821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "cfab11ee08c84aed9f041d96cd10720f"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "6f3d85e419af16990f09df9c2d13531d"
server_key_iv: "fca682f6a71f1ad8994104db76cdc063"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.372294 3821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:27:09.375607 3836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.377100 3821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:27:09.377274 3821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "cfab11ee08c84aed9f041d96cd10720f"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "6f3d85e419af16990f09df9c2d13531d"
server_key_iv: "fca682f6a71f1ad8994104db76cdc063"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.377532 3821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:09.397113 3821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:09.401170 3821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:09.401494 3821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:09.402370 3821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:09.403750 3821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:09.403820 3821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.403904 3821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:09.403949 3821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.436815 3821 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:45981
I20260926 19:27:09.436905 3948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:45981 every 8 connection(s)
I20260926 19:27:09.438210 3821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:09.443926 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3821
I20260926 19:27:09.444128 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:09.444633 1180 external_mini_cluster.cc:1428] Setting key 4517afce33853cb32523f5b607397937
Sep 26 19:27:09 dist-test-slave-9wb0 krb5kdc[3581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:09.459542 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:09.440770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:55547 (local address 127.1.39.62:40587)
0926 19:27:09.440966 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:09.440972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:09.442636 (+ 1664us) server_negotiation.cc:407] Connection header received
0926 19:27:09.444524 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:09.444530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:09.444614 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:09.444737 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:09.446707 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.447715 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.449214 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.449961 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.450113 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:09.453678 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:09.453683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:09.453686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:09.453716 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:09.456229 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:09.457196 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:09.457199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:09.457202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:09.457383 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:09.457852 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:09.457855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:09.457857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:09.458114 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:09.458229 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:09.459138 (+ 909us) server_negotiation.cc:299] Negotiation successful
0926 19:27:09.459301 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:27:09.461167 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:09.441421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40587 (local address 127.1.39.2:55547)
0926 19:27:09.442410 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:09.442514 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:09.444240 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:09.444975 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:09.444984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:09.445758 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:09.446524 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:09.446554 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.447868 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.447872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:09.449070 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:09.449082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.450177 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.450183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:09.450372 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:09.451071 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:09.451097 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:09.453472 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:09.456372 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:09.456383 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:09.456402 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:09.457071 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:09.457492 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:09.457500 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:09.457506 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:09.457741 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:09.458243 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:09.458254 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:09.458734 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 19:27:09.459041 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 19:27:09.459652 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":360,"threads_started":1}
I20260926 19:27:09.462685 3949 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40587
I20260926 19:27:09.463040 3949 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:09.464171 3949 heartbeater.cc:511] Master 127.1.39.62:40587 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:27:09.465945 3626 ts_manager.cc:194] Registered new tserver with Master: cfab11ee08c84aed9f041d96cd10720f (127.1.39.2:45981)
I20260926 19:27:09.466804 3626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:55547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:09.492468 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:09.812879 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:09.813207 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:09.813398 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:09.820561 3954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:09.820713 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:09.820904 3954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:09.830631 3954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:09.831913 3954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:09.833226 3954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:09.847361 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:09.847365 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:09.847404 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:09.848040 3954 server_base.cc:1034] running on GCE node
I20260926 19:27:09.848713 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:09.849786 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:09.851013 3954 hybrid_clock.cc:648] HybridClock initialized: now 1790450829850963 us; error 51 us; skew 500 ppm
I20260926 19:27:09.851329 3954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:09 dist-test-slave-9wb0 krb5kdc[3581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:09.854561 3954 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:09.855914 3954 webserver.cc:460] Webserver started at http://127.1.39.3:35841/ using document root <none> and password file <none>
I20260926 19:27:09.856537 3954 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:09.856643 3954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:09.856899 3954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:09.859737 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "4144c35b7b9646ae8564f919bee8a8b0"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "26e3c35186c2e813db366ff8751bd9d4"
server_key_iv: "8464549a4149a4c492b56c2f5561af94"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.860431 3954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "4144c35b7b9646ae8564f919bee8a8b0"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "26e3c35186c2e813db366ff8751bd9d4"
server_key_iv: "8464549a4149a4c492b56c2f5561af94"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.865808 3954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 19:27:09.869207 3969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.870602 3954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:09.870792 3954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "4144c35b7b9646ae8564f919bee8a8b0"
format_stamp: "Formatted at 2026-09-26 19:27:09 on dist-test-slave-9wb0"
server_key: "26e3c35186c2e813db366ff8751bd9d4"
server_key_iv: "8464549a4149a4c492b56c2f5561af94"
server_key_version: "encryptionkey@0"
I20260926 19:27:09.870980 3954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:09.886353 3954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:09.890733 3954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:09.891026 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:09.892042 3954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:09.893530 3954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:09.893602 3954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.893739 3954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:09.893791 3954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:09.926014 3954 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:34409
I20260926 19:27:09.926056 4081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:34409 every 8 connection(s)
I20260926 19:27:09.927356 3954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:09.930027 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 3954
I20260926 19:27:09.930243 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:09.930745 1180 external_mini_cluster.cc:1428] Setting key 0cc9e97bace8c239f11c45d25f31f3fe
Sep 26 19:27:09 dist-test-slave-9wb0 krb5kdc[3581](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450829, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:09.951351 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:09.929946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:47415 (local address 127.1.39.62:40587)
0926 19:27:09.930155 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:09.930161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:09.931709 (+ 1548us) server_negotiation.cc:407] Connection header received
0926 19:27:09.934152 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:09.934158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:09.934252 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:09.934370 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:09.935891 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.936664 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.938306 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.938694 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.938833 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:09.944512 (+ 5679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:09.944519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:09.944524 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:09.944562 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:09.948170 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:09.949111 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:09.949115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:09.949117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:09.949248 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:09.949783 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:09.949787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:09.949789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:09.950040 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:09.950174 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:09.950957 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 19:27:09.951127 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:27:09.952647 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:09.930667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40587 (local address 127.1.39.3:47415)
0926 19:27:09.931527 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:09.931589 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:09.933885 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:09.934618 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:09.934629 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:09.935151 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:09.935725 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:09.935744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.936815 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.936820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:09.938156 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:09.938168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:09.938930 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:09.938936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:09.939286 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:09.941493 (+ 2207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:09.941519 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:09.944260 (+ 2741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:09.948350 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:09.948360 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:09.948377 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:09.948962 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:09.949417 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:09.949424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:09.949429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:09.949674 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:09.950176 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:09.950184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:09.950565 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 19:27:09.950868 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 19:27:09.951391 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":316,"threads_started":1}
I20260926 19:27:09.954186 4082 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40587
I20260926 19:27:09.954506 4082 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:09.955246 4082 heartbeater.cc:511] Master 127.1.39.62:40587 requested a full tablet report, sending...
I20260926 19:27:09.956812 3626 ts_manager.cc:194] Registered new tserver with Master: 4144c35b7b9646ae8564f919bee8a8b0 (127.1.39.3:34409)
I20260926 19:27:09.957672 3626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:47415
I20260926 19:27:09.960716 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:27:09.975847 3815 heartbeater.cc:503] Master 127.1.39.62:40587 was elected leader, sending a full tablet report...
W20260926 19:27:09.992683 3817 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:09.989916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.1.39.62:40587)
0926 19:27:09.990086 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:09.990090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:09.990702 (+ 612us) server_negotiation.cc:407] Connection header received
0926 19:27:09.990859 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:09.990863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:09.990926 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:09.991029 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:09.992264 (+ 1235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60472: BlockingRecv error: recv got EOF from 127.0.0.1:60472 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:27:09.996186 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3687
W20260926 19:27:09.999048 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3687: Not authorized: Client connection negotiation failed: client connection to 127.1.39.1:33143: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:27:09.999236 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3687
I20260926 19:27:10.012143 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3821
W20260926 19:27:10.015188 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3821: Not authorized: Client connection negotiation failed: client connection to 127.1.39.2:45981: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:27:10.015338 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3821
I20260926 19:27:10.032353 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3954
W20260926 19:27:10.035343 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3954: Not authorized: Client connection negotiation failed: client connection to 127.1.39.3:34409: server requires authentication, but client does not have Kerberos credentials available
I20260926 19:27:10.035462 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3954
I20260926 19:27:10.050845 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 3594
I20260926 19:27:10.052109 3656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:10.156121 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 3594
2026-09-26T19:27:10Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3585 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:10 dist-test-slave-9wb0 krb5kdc[4099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:10 dist-test-slave-9wb0 krb5kdc[4099](info): set up 2 sockets
Sep 26 19:27:10 dist-test-slave-9wb0 krb5kdc[4099](info): commencing operation
krb5kdc: starting...
W20260926 19:27:11.273947 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:11 dist-test-slave-9wb0 krb5kdc[4099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:11.481858 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:35687
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:35687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:11.805706 4112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:11.805994 4112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:11.806093 4112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:11.813263 4112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:11.813436 4112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:11.813484 4112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:11.813542 4112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:11.813583 4112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:11.823385 4112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:35687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:35687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:11.824625 4112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:11.825912 4112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:11.837549 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:11.837510 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:11.837584 4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:11.838701 4112 server_base.cc:1034] running on GCE node
I20260926 19:27:11.839496 4112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:11.840821 4112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:11.842059 4112 hybrid_clock.cc:648] HybridClock initialized: now 1790450831842011 us; error 60 us; skew 500 ppm
I20260926 19:27:11.842399 4112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:11 dist-test-slave-9wb0 krb5kdc[4099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450831, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:11.845394 4112 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:11.846689 4112 webserver.cc:460] Webserver started at http://127.1.39.62:44275/ using document root <none> and password file <none>
I20260926 19:27:11.847303 4112 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:11.847397 4112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:11.847635 4112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:11.850533 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "f93f786452334eef9548a030716216a0"
format_stamp: "Formatted at 2026-09-26 19:27:11 on dist-test-slave-9wb0"
server_key: "7ba96edf64fa3b502729d417d8ea45c0"
server_key_iv: "3c63d4ea32734ec53a98b66beb8a5804"
server_key_version: "encryptionkey@0"
I20260926 19:27:11.851234 4112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "f93f786452334eef9548a030716216a0"
format_stamp: "Formatted at 2026-09-26 19:27:11 on dist-test-slave-9wb0"
server_key: "7ba96edf64fa3b502729d417d8ea45c0"
server_key_iv: "3c63d4ea32734ec53a98b66beb8a5804"
server_key_version: "encryptionkey@0"
I20260926 19:27:11.856050 4112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 19:27:11.859277 4127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:11.860625 4112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:27:11.860800 4112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "f93f786452334eef9548a030716216a0"
format_stamp: "Formatted at 2026-09-26 19:27:11 on dist-test-slave-9wb0"
server_key: "7ba96edf64fa3b502729d417d8ea45c0"
server_key_iv: "3c63d4ea32734ec53a98b66beb8a5804"
server_key_version: "encryptionkey@0"
I20260926 19:27:11.861001 4112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:11.878676 4112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:11.882812 4112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:11.883106 4112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:11.901196 4112 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:35687
I20260926 19:27:11.901243 4178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:35687 every 8 connection(s)
I20260926 19:27:11.902578 4112 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:11.906466 4179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:11.911053 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4112
I20260926 19:27:11.911247 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:11.912091 1180 external_mini_cluster.cc:1428] Setting key 518344f54ed0117a0d03fe3df2c06fea
I20260926 19:27:11.918277 4179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0: Bootstrap starting.
I20260926 19:27:11.921909 4179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:11.923139 4179 log.cc:824] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:11.926606 4179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0: No bootstrap required, opened a new log
Sep 26 19:27:11 dist-test-slave-9wb0 krb5kdc[4099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:11.934952 4179 raft_consensus.cc:348] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } }
I20260926 19:27:11.935372 4179 raft_consensus.cc:374] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:11.935534 4179 raft_consensus.cc:729] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93f786452334eef9548a030716216a0, State: Initialized, Role: FOLLOWER
I20260926 19:27:11.935992 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:11.913991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.1.39.62:35687)
0926 19:27:11.914790 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:11.914802 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:11.914844 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:27:11.916684 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:11.916707 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:11.917100 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:11.917486 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:11.919251 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:11.920546 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:11.922341 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:11.923036 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:11.923179 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:11.928146 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:11.928161 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:11.928181 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:11.928226 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:11.931626 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:11.932370 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:11.932377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:11.932384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:11.932575 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:11.932958 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:11.932962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:11.932964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:11.933631 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:11.933951 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:11.934398 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 19:27:11.934822 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":369,"threads_started":1}
I20260926 19:27:11.936256 4179 consensus_queue.cc:260] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } }
I20260926 19:27:11.936506 4179 raft_consensus.cc:388] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:11.936661 4179 raft_consensus.cc:482] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:11.936859 4179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:11.940199 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:27:11.940131 4179 raft_consensus.cc:504] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } }
I20260926 19:27:11.940623 4179 leader_election.cc:302] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f93f786452334eef9548a030716216a0; no voters:
I20260926 19:27:11.941321 4179 leader_election.cc:288] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:11.941430 4184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:11.943889 4179 sys_catalog.cc:564] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:11.945595 4184 raft_consensus.cc:686] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [term 1 LEADER]: Becoming Leader. State: Replica: f93f786452334eef9548a030716216a0, State: Running, Role: LEADER
I20260926 19:27:11.946388 4184 consensus_queue.cc:237] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } }
I20260926 19:27:11.949836 4186 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f93f786452334eef9548a030716216a0. Latest consensus state: current_term: 1 leader_uuid: "f93f786452334eef9548a030716216a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } } }
I20260926 19:27:11.949939 4185 sys_catalog.cc:455] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f93f786452334eef9548a030716216a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f93f786452334eef9548a030716216a0" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35687 } } }
I20260926 19:27:11.950274 4186 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:11.950337 4185 sys_catalog.cc:458] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:11.962363 4192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:11.968587 4192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:11.995621 4192 catalog_manager.cc:1347] Generated new cluster ID: 4bf3116f91174ea792a9d86a6d0194c6
I20260926 19:27:11.995764 4192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:12.007635 4192 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:12.009651 4192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:12.029716 4192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f93f786452334eef9548a030716216a0: Generated new TSK 0
I20260926 19:27:12.030860 4192 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:27:12.032946 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:12.023887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.1.39.62:35687)
0926 19:27:12.024087 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:12.024095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:12.024675 (+ 580us) server_negotiation.cc:407] Connection header received
0926 19:27:12.024916 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:12.024928 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:12.025002 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:12.025127 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:12.026343 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:12.027587 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:12.029613 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:12.030102 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:12.030264 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:12.031420 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:12.031425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:12.031428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:12.032135 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:12.032214 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:12.032245 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:12.032370 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:12.032510 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 19:27:12.032610 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:27:12.040194 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4112
I20260926 19:27:12.042378 4174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:12.132301 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4112
2026-09-26T19:27:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1967 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:12 dist-test-slave-9wb0 krb5kdc[4213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:12 dist-test-slave-9wb0 krb5kdc[4213](info): set up 2 sockets
Sep 26 19:27:12 dist-test-slave-9wb0 krb5kdc[4213](info): commencing operation
krb5kdc: starting...
W20260926 19:27:13.233872 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:13 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:13.432941 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:43001
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:43001
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:13.761966 4226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:13.762319 4226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:13.762486 4226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:13.769603 4226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:13.769753 4226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:13.769860 4226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:13.769961 4226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:13.770043 4226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:13.779918 4226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:43001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:43001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:13.781311 4226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:13.782688 4226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:13.794528 4231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:13.794467 4232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:13.794415 4234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:13.795116 4226 server_base.cc:1034] running on GCE node
I20260926 19:27:13.795923 4226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:13.797349 4226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:13.798556 4226 hybrid_clock.cc:648] HybridClock initialized: now 1790450833798517 us; error 30 us; skew 500 ppm
I20260926 19:27:13.798902 4226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:13 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450833, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:13.801961 4226 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:13.803447 4226 webserver.cc:460] Webserver started at http://127.1.39.62:39121/ using document root <none> and password file <none>
I20260926 19:27:13.804126 4226 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:13.804267 4226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:13.804569 4226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:13.807516 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "7caf1614b811458bb3e0d1e64434817a"
format_stamp: "Formatted at 2026-09-26 19:27:13 on dist-test-slave-9wb0"
server_key: "1f5d2ed2f9b4443ab7862f7396f209a5"
server_key_iv: "b15dd1f0e42031f9e9f8edf23979f03f"
server_key_version: "encryptionkey@0"
I20260926 19:27:13.808259 4226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "7caf1614b811458bb3e0d1e64434817a"
format_stamp: "Formatted at 2026-09-26 19:27:13 on dist-test-slave-9wb0"
server_key: "1f5d2ed2f9b4443ab7862f7396f209a5"
server_key_iv: "b15dd1f0e42031f9e9f8edf23979f03f"
server_key_version: "encryptionkey@0"
I20260926 19:27:13.813443 4226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 19:27:13.816658 4241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:13.817927 4226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:13.818111 4226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "7caf1614b811458bb3e0d1e64434817a"
format_stamp: "Formatted at 2026-09-26 19:27:13 on dist-test-slave-9wb0"
server_key: "1f5d2ed2f9b4443ab7862f7396f209a5"
server_key_iv: "b15dd1f0e42031f9e9f8edf23979f03f"
server_key_version: "encryptionkey@0"
I20260926 19:27:13.818305 4226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:13.833509 4226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:13.837327 4226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:13.837647 4226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:13.856137 4226 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:43001
I20260926 19:27:13.856165 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:43001 every 8 connection(s)
I20260926 19:27:13.857589 4226 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:13.860883 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4226
I20260926 19:27:13.861102 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:13.861704 4293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:13.861948 1180 external_mini_cluster.cc:1428] Setting key 357704f8d39e6e109dac0559bcd8238f
I20260926 19:27:13.876786 4293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a: Bootstrap starting.
Sep 26 19:27:13 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:13.880085 4293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:13.881146 4293 log.cc:824] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:13.884016 4293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a: No bootstrap required, opened a new log
I20260926 19:27:13.885737 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:13.864022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.1.39.62:43001)
0926 19:27:13.864860 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:13.864872 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:13.864912 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:27:13.866575 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:13.866599 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:13.867006 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:13.867343 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:13.869172 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:13.870649 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:13.872459 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:13.873231 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:13.873497 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:13.878301 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:13.878310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:13.878329 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:13.878378 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:13.881586 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:13.882332 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:13.882340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:13.882346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:13.882525 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:13.882916 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:13.882922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:13.882925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:13.883458 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:13.883779 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:13.884214 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 19:27:13.884662 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":394,"threads_started":1}
I20260926 19:27:13.892285 4293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } }
I20260926 19:27:13.892673 4293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:13.892781 4293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caf1614b811458bb3e0d1e64434817a, State: Initialized, Role: FOLLOWER
I20260926 19:27:13.893487 4293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } }
I20260926 19:27:13.893769 4293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:13.893929 4293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:13.894129 4293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:13.896296 4293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } }
I20260926 19:27:13.896814 4293 leader_election.cc:302] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caf1614b811458bb3e0d1e64434817a; no voters:
I20260926 19:27:13.897504 4293 leader_election.cc:288] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:13.897636 4298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:13.898810 4298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [term 1 LEADER]: Becoming Leader. State: Replica: 7caf1614b811458bb3e0d1e64434817a, State: Running, Role: LEADER
I20260926 19:27:13.899355 4298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } }
I20260926 19:27:13.899698 4293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:13.902245 4300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caf1614b811458bb3e0d1e64434817a. Latest consensus state: current_term: 1 leader_uuid: "7caf1614b811458bb3e0d1e64434817a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } } }
I20260926 19:27:13.902640 4300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:13.902379 4299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7caf1614b811458bb3e0d1e64434817a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caf1614b811458bb3e0d1e64434817a" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 43001 } } }
I20260926 19:27:13.902807 4299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:13.906158 4306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:13.912297 4306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:13.920917 4306 catalog_manager.cc:1347] Generated new cluster ID: 34a7ea6b4751457f9c31b41139dc902c
I20260926 19:27:13.921109 4306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:13.931931 4306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:13.933260 4306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:13.940300 4306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7caf1614b811458bb3e0d1e64434817a: Generated new TSK 0
I20260926 19:27:13.941618 4306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:13.991084 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:14.318845 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:14.319116 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:14.319212 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:14.326452 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:14.326557 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:14.326670 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:14.336346 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:14.337719 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:14.339114 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:14.352499 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:14.352440 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:14.352468 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:14.353598 4319 server_base.cc:1034] running on GCE node
I20260926 19:27:14.354133 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:14.355021 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:14.356217 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790450834356185 us; error 31 us; skew 500 ppm
I20260926 19:27:14.356492 4319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:14 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:14.359643 4319 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:14.361110 4319 webserver.cc:460] Webserver started at http://127.1.39.1:45015/ using document root <none> and password file <none>
I20260926 19:27:14.361881 4319 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:14.362030 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:14.362336 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:14.365175 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "893b979909924b8491d893a33f03ccfe"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "919b7550ab7c79bf371267dbc45ec384"
server_key_iv: "abfe3018c2a2cba6b85f3361100130c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.366035 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "893b979909924b8491d893a33f03ccfe"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "919b7550ab7c79bf371267dbc45ec384"
server_key_iv: "abfe3018c2a2cba6b85f3361100130c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.371044 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:27:14.374353 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.375633 4319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260926 19:27:14.375800 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "893b979909924b8491d893a33f03ccfe"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "919b7550ab7c79bf371267dbc45ec384"
server_key_iv: "abfe3018c2a2cba6b85f3361100130c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.375993 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:14.407140 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:14.411603 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:14.412029 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:14.413074 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:14.414469 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:14.414541 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.414623 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:14.414666 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.448273 4319 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33131
I20260926 19:27:14.448378 4446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33131 every 8 connection(s)
I20260926 19:27:14.449697 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:14.450207 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4319
I20260926 19:27:14.450400 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:14.451243 1180 external_mini_cluster.cc:1428] Setting key bbb15f7a815653951d384df1ee74e9ae
Sep 26 19:27:14 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:27:14.472054 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:14.452449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:33361 (local address 127.1.39.62:43001)
0926 19:27:14.452954 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:14.452961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:14.454177 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 19:27:14.455973 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:14.455981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:14.456073 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:14.456193 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:14.457710 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.458911 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.460429 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.461197 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.461371 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:14.464902 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:14.464912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:14.464919 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:14.464963 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:14.468717 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:14.469531 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:14.469540 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:14.469545 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:14.469696 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:14.470119 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:14.470124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:14.470128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:14.470519 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:14.470676 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:14.471367 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 19:27:14.471638 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260926 19:27:14.472719 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:14.453128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:43001 (local address 127.1.39.1:33361)
0926 19:27:14.454003 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:14.454056 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:14.455702 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:14.456391 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:14.456400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:14.456884 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:14.457533 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:14.457552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.459074 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.459078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:14.460279 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:14.460290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.461453 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.461459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:14.461639 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:14.462316 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:14.462337 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:14.464634 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:14.468861 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:14.468867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:14.468881 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:14.469359 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:14.469809 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:14.469812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:14.469815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:14.469984 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:14.470653 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:14.470659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:14.470954 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 19:27:14.471193 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 19:27:14.471578 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":402,"threads_started":1}
I20260926 19:27:14.474303 4447 heartbeater.cc:347] Connected to a master server at 127.1.39.62:43001
I20260926 19:27:14.474617 4447 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
I20260926 19:27:14.475382 4447 heartbeater.cc:511] Master 127.1.39.62:43001 requested a full tablet report, sending...
I20260926 19:27:14.477535 4258 ts_manager.cc:194] Registered new tserver with Master: 893b979909924b8491d893a33f03ccfe (127.1.39.1:33131)
I20260926 19:27:14.479089 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:33361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:14.504104 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:14.823390 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:14.823714 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:14.823879 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:14.831676 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:14.831815 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:14.831982 4453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:14.841862 4453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:14.843163 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:14.844460 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:14.859476 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:14.859596 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:14.859979 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:14.860345 4453 server_base.cc:1034] running on GCE node
I20260926 19:27:14.860937 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:14.861930 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:14.863152 4453 hybrid_clock.cc:648] HybridClock initialized: now 1790450834863087 us; error 59 us; skew 500 ppm
I20260926 19:27:14.863468 4453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:14 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:14.866756 4453 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:14.868324 4453 webserver.cc:460] Webserver started at http://127.1.39.2:33497/ using document root <none> and password file <none>
I20260926 19:27:14.869000 4453 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:14.869125 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:14.869475 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:14.872520 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "4f2340b403ed465db2bb34d17dc2feaf"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "35ba511d78691cebef825de32c32e2e4"
server_key_iv: "4ead5024ef9880d1c0d195e384798341"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.873370 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "4f2340b403ed465db2bb34d17dc2feaf"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "35ba511d78691cebef825de32c32e2e4"
server_key_iv: "4ead5024ef9880d1c0d195e384798341"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.878537 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:27:14.882195 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.883600 4453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:14.883762 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "4f2340b403ed465db2bb34d17dc2feaf"
format_stamp: "Formatted at 2026-09-26 19:27:14 on dist-test-slave-9wb0"
server_key: "35ba511d78691cebef825de32c32e2e4"
server_key_iv: "4ead5024ef9880d1c0d195e384798341"
server_key_version: "encryptionkey@0"
I20260926 19:27:14.883937 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:14.901876 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:14.905910 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:14.906214 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:14.907155 4453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:14.908442 4453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:14.908512 4453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.908677 4453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:14.908735 4453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:14.942098 4453 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:43975
I20260926 19:27:14.942144 4580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:43975 every 8 connection(s)
I20260926 19:27:14.943388 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:14.953119 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4453
I20260926 19:27:14.953330 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:14.953805 1180 external_mini_cluster.cc:1428] Setting key 1f907b37524336c1c5a877c90618c8ce
Sep 26 19:27:14 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450834, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:14.963774 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:14.945990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:41261 (local address 127.1.39.62:43001)
0926 19:27:14.946178 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:14.946183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:14.947509 (+ 1326us) server_negotiation.cc:407] Connection header received
0926 19:27:14.949259 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:14.949265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:14.949377 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:14.949497 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:14.950956 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.951677 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.953556 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.954030 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.954142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:14.958008 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:14.958013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:14.958016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:14.958052 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:14.960572 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:14.961280 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:14.961288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:14.961320 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:14.961454 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:14.961904 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:14.961909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:14.961913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:14.962340 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:14.962531 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:14.963210 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 19:27:14.963424 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:27:14.964766 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:14.946475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:43001 (local address 127.1.39.2:41261)
0926 19:27:14.947308 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:14.947355 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:14.948995 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:14.949730 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:14.949738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:14.950252 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:14.950791 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:14.950807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.951830 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.951835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:14.953356 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:14.953373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:14.954259 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:14.954266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:14.954512 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:14.955271 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:14.955296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:14.957777 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:14.960717 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:14.960726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:14.960743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:14.961158 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:14.961566 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:14.961573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:14.961578 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:14.961762 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:14.962505 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:14.962513 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:14.962847 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 19:27:14.963144 (+ 297us) client_negotiation.cc:240] Negotiation successful
0926 19:27:14.963661 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":300,"threads_started":1}
I20260926 19:27:14.966349 4581 heartbeater.cc:347] Connected to a master server at 127.1.39.62:43001
I20260926 19:27:14.966645 4581 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:14.967368 4581 heartbeater.cc:511] Master 127.1.39.62:43001 requested a full tablet report, sending...
I20260926 19:27:14.968799 4258 ts_manager.cc:194] Registered new tserver with Master: 4f2340b403ed465db2bb34d17dc2feaf (127.1.39.2:43975)
I20260926 19:27:14.969623 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:41261
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:14.997414 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:15.318534 4586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:15.318854 4586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:15.319022 4586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:15.326354 4586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:15.326494 4586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:15.326678 4586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:15.336328 4586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:43001
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:15.337769 4586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:15.339077 4586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:15.351785 4592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:15.352285 4591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:15.352576 4594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:15.352708 4586 server_base.cc:1034] running on GCE node
I20260926 19:27:15.353407 4586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:15.354370 4586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:15.355614 4586 hybrid_clock.cc:648] HybridClock initialized: now 1790450835355555 us; error 49 us; skew 500 ppm
I20260926 19:27:15.355945 4586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:15.359004 4586 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:15.360316 4586 webserver.cc:460] Webserver started at http://127.1.39.3:40969/ using document root <none> and password file <none>
I20260926 19:27:15.360992 4586 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:15.361132 4586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:15.361498 4586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:15.364845 4586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "42fb27e2d65f4ed0b691fcacb804218c"
format_stamp: "Formatted at 2026-09-26 19:27:15 on dist-test-slave-9wb0"
server_key: "ca81961199854d5dad89c02ed6f10a3f"
server_key_iv: "07219912e1f6b26ed19872ec4483a657"
server_key_version: "encryptionkey@0"
I20260926 19:27:15.365713 4586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "42fb27e2d65f4ed0b691fcacb804218c"
format_stamp: "Formatted at 2026-09-26 19:27:15 on dist-test-slave-9wb0"
server_key: "ca81961199854d5dad89c02ed6f10a3f"
server_key_iv: "07219912e1f6b26ed19872ec4483a657"
server_key_version: "encryptionkey@0"
I20260926 19:27:15.370642 4586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:27:15.374192 4601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:15.375789 4586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:15.375923 4586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "42fb27e2d65f4ed0b691fcacb804218c"
format_stamp: "Formatted at 2026-09-26 19:27:15 on dist-test-slave-9wb0"
server_key: "ca81961199854d5dad89c02ed6f10a3f"
server_key_iv: "07219912e1f6b26ed19872ec4483a657"
server_key_version: "encryptionkey@0"
I20260926 19:27:15.376098 4586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:15.412047 4586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:15.415982 4586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:15.416301 4586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:15.417234 4586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:15.418540 4586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:15.418610 4586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:15.418699 4586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:15.418741 4586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:15.451598 4586 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38269
I20260926 19:27:15.451618 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38269 every 8 connection(s)
I20260926 19:27:15.452873 4586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:15.456728 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4586
I20260926 19:27:15.456923 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:15.457562 1180 external_mini_cluster.cc:1428] Setting key e0abbc3bb3af677787a3ea04fcdb2015
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:15.473522 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.455518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:49851 (local address 127.1.39.62:43001)
0926 19:27:15.455717 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:15.455722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:15.457010 (+ 1288us) server_negotiation.cc:407] Connection header received
0926 19:27:15.459312 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:15.459318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:15.459401 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:15.459526 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:15.461234 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.462052 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.463533 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.463926 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.464031 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.467490 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:15.467495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:15.467498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:15.467528 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:15.470080 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.470771 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.470775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.470777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.470921 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.471429 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.471435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.471438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.471719 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:15.471990 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:15.472906 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 19:27:15.473185 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:27:15.474494 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.455989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:43001 (local address 127.1.39.3:49851)
0926 19:27:15.456827 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:15.456888 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:15.459008 (+ 2120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:15.459815 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:15.459823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:15.460402 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:15.461079 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:15.461096 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.462180 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.462183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:15.463374 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:15.463385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.464124 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.464128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:15.464311 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.464989 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:15.465016 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:15.467274 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:15.470202 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:15.470208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:15.470221 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:15.470634 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:15.471044 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:15.471049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:15.471052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:15.471262 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:15.471930 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:15.471958 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:15.472429 (+ 471us) client_negotiation.cc:769] Sending connection context
0926 19:27:15.472747 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 19:27:15.473291 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":340,"threads_started":1}
I20260926 19:27:15.475962 4714 heartbeater.cc:347] Connected to a master server at 127.1.39.62:43001
I20260926 19:27:15.476291 4714 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:15.477022 4714 heartbeater.cc:511] Master 127.1.39.62:43001 requested a full tablet report, sending...
I20260926 19:27:15.478535 4258 ts_manager.cc:194] Registered new tserver with Master: 42fb27e2d65f4ed0b691fcacb804218c (127.1.39.3:38269)
I20260926 19:27:15.479374 4258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:49851
I20260926 19:27:15.482089 4447 heartbeater.cc:503] Master 127.1.39.62:43001 was elected leader, sending a full tablet report...
I20260926 19:27:15.485338 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:15.535358 4449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.519849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48914 (local address 127.1.39.62:43001)
0926 19:27:15.520028 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:15.520034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:15.520541 (+ 507us) server_negotiation.cc:407] Connection header received
0926 19:27:15.520688 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:15.520692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:15.520753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:15.520846 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:15.522309 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.523012 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.524210 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.524605 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.524701 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.529919 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:15.529930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:15.529935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:15.530000 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:15.532897 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.533696 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.533700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.533702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.533827 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.534182 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.534185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.534187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.534511 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:15.534686 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:15.534978 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 19:27:15.535137 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 19:27:15.535986 4258 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48914
I20260926 19:27:15.538224 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4319
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:15.554100 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.539578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.1.39.1:33131)
0926 19:27:15.540184 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:15.540192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:15.540216 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:27:15.540275 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:15.540281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:15.540479 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:15.540619 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:15.541743 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.542715 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.544144 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.544707 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.544801 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.548786 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:15.548797 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:15.548811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:15.548845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:15.551648 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.552227 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.552234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.552236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.552351 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.552678 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.552682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.552683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.553009 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:15.553231 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:15.553505 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 19:27:15.553691 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":344,"threads_started":1}
W20260926 19:27:15.555317 4442 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55360
W20260926 19:27:15.556406 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4319: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:15.556586 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4319
I20260926 19:27:15.572157 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4453
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:15.587196 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.572947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46888 (local address 127.1.39.2:43975)
0926 19:27:15.573574 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:15.573581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:15.573607 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:27:15.573672 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:15.573679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:15.573878 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:15.574026 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:15.575166 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.576150 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.577350 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.577884 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.577993 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.582084 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:15.582094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:15.582109 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:15.582143 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:15.584656 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.585212 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.585218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.585220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.585445 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.585766 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.585769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.585771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.586082 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:15.586285 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:15.586604 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:27:15.586857 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":338,"threads_started":1}
W20260926 19:27:15.588838 4576 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46888
W20260926 19:27:15.589980 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4453: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:15.590143 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4453
I20260926 19:27:15.605937 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4586
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450835, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:15.621450 4727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:15.606957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.1.39.3:38269)
0926 19:27:15.607578 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:15.607585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:15.607612 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:27:15.607667 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:15.607673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:15.607872 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:15.608044 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:15.609368 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.610403 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.611629 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:15.612178 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:15.612319 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:15.616415 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:15.616425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:15.616440 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:15.616475 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:15.619024 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.619619 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.619626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.619628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.619737 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:15.620021 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:15.620025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:15.620026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:15.620329 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:15.620535 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:15.620791 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 19:27:15.620988 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
W20260926 19:27:15.622814 4709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:57886
W20260926 19:27:15.623814 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4586: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:27:15.624008 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4586
I20260926 19:27:15.639401 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4226
I20260926 19:27:15.640497 4288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:15.742043 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4226
2026-09-26T19:27:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3614 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4731](info): set up 2 sockets
Sep 26 19:27:15 dist-test-slave-9wb0 krb5kdc[4731](info): commencing operation
krb5kdc: starting...
W20260926 19:27:16.825819 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:16 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:17.019075 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38561
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:17.333189 4744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:17.333487 4744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:17.333688 4744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:17.340842 4744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:17.340956 4744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:17.341045 4744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:17.341131 4744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:17.341213 4744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:17.351035 4744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:17.352391 4744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:17.353762 4744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:17.365661 4750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:17.365661 4752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:17.365695 4749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:17.366385 4744 server_base.cc:1034] running on GCE node
I20260926 19:27:17.367230 4744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:17.368629 4744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:17.369850 4744 hybrid_clock.cc:648] HybridClock initialized: now 1790450837369809 us; error 61 us; skew 500 ppm
I20260926 19:27:17.370204 4744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:17 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:17.373126 4744 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:17.374503 4744 webserver.cc:460] Webserver started at http://127.1.39.62:40573/ using document root <none> and password file <none>
I20260926 19:27:17.375178 4744 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:17.375319 4744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:17.375643 4744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:17.378497 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "cd2cceafc62743b99f9b1ee9f38afb21"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "a9fb9bb8ebfd4faa92ee8c36b9f8e0c6"
server_key_iv: "0d8021cda59361a4cbcba88fc7803719"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.379257 4744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "cd2cceafc62743b99f9b1ee9f38afb21"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "a9fb9bb8ebfd4faa92ee8c36b9f8e0c6"
server_key_iv: "0d8021cda59361a4cbcba88fc7803719"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.384332 4744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260926 19:27:17.387578 4759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:17.388782 4744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:27:17.388955 4744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cd2cceafc62743b99f9b1ee9f38afb21"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "a9fb9bb8ebfd4faa92ee8c36b9f8e0c6"
server_key_iv: "0d8021cda59361a4cbcba88fc7803719"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.389143 4744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:17.408951 4744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:17.412730 4744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:17.413017 4744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:17.437970 4744 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:38561
I20260926 19:27:17.438032 4810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:38561 every 8 connection(s)
I20260926 19:27:17.439930 4744 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:17.445421 4811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:17.447292 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4744
I20260926 19:27:17.447540 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:17.448341 1180 external_mini_cluster.cc:1428] Setting key 83d1b192c1d76580b8c4a61c93d2caec
I20260926 19:27:17.460687 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21: Bootstrap starting.
I20260926 19:27:17.464123 4811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:17.465447 4811 log.cc:824] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:27:17 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:17.469161 4811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21: No bootstrap required, opened a new log
I20260926 19:27:17.476926 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:17.450541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.1.39.62:38561)
0926 19:27:17.451433 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:17.451451 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:17.451502 (+ 51us) server_negotiation.cc:407] Connection header received
0926 19:27:17.454178 (+ 2676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:17.454215 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:17.454806 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:17.455313 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:17.457123 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:17.459571 (+ 2448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:17.461825 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:17.463089 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:17.463308 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:17.468481 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:17.468495 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:17.468538 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:17.468595 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:17.472900 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:17.473746 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:17.473752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:17.473758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:17.473931 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:17.474329 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:17.474332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:17.474334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:17.474822 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:17.475023 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:17.475466 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 19:27:17.475852 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":364,"threads_started":1}
I20260926 19:27:17.478667 4811 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } }
I20260926 19:27:17.479040 4811 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:17.479166 4811 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2cceafc62743b99f9b1ee9f38afb21, State: Initialized, Role: FOLLOWER
I20260926 19:27:17.479799 4811 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } }
I20260926 19:27:17.480089 4811 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:17.480242 4811 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:17.480422 4811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:17.482564 4811 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } }
I20260926 19:27:17.483143 4811 leader_election.cc:302] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2cceafc62743b99f9b1ee9f38afb21; no voters:
I20260926 19:27:17.483768 4811 leader_election.cc:288] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:17.483856 4816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:17.484663 4816 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [term 1 LEADER]: Becoming Leader. State: Replica: cd2cceafc62743b99f9b1ee9f38afb21, State: Running, Role: LEADER
I20260926 19:27:17.485226 4816 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } }
I20260926 19:27:17.485898 4811 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:17.488052 4818 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2cceafc62743b99f9b1ee9f38afb21. Latest consensus state: current_term: 1 leader_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } } }
I20260926 19:27:17.488462 4818 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:17.489552 4817 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2cceafc62743b99f9b1ee9f38afb21" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38561 } } }
I20260926 19:27:17.489998 4817 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:17.492522 4825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:17.497663 4825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:17.505555 4825 catalog_manager.cc:1347] Generated new cluster ID: 38a6e479d47d42c2881f11511b225765
I20260926 19:27:17.505681 4825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:17.531651 4825 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:17.532989 4825 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:17.542383 4825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2cceafc62743b99f9b1ee9f38afb21: Generated new TSK 0
I20260926 19:27:17.543239 4825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:17.592651 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:17.916692 4837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:17.916980 4837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:17.917073 4837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:17.924376 4837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:17.924487 4837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:17.924598 4837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:17.934542 4837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:17.935832 4837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:17.937173 4837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:17.950546 4842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:17.950620 4845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:17.950578 4843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:17.951918 4837 server_base.cc:1034] running on GCE node
I20260926 19:27:17.952473 4837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:17.953361 4837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:17.954603 4837 hybrid_clock.cc:648] HybridClock initialized: now 1790450837954550 us; error 45 us; skew 500 ppm
I20260926 19:27:17.954876 4837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:17 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:17.957935 4837 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:17.959331 4837 webserver.cc:460] Webserver started at http://127.1.39.1:37797/ using document root <none> and password file <none>
I20260926 19:27:17.960000 4837 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:17.960134 4837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:17.960445 4837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:17.963388 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "1b1c017649a3428a8d0a5c48663daf7d"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "b6b399a51c9eefaac713ad11d4dc9356"
server_key_iv: "544d3a4accdba71b9ae43914cfdd666b"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.964250 4837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1c017649a3428a8d0a5c48663daf7d"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "b6b399a51c9eefaac713ad11d4dc9356"
server_key_iv: "544d3a4accdba71b9ae43914cfdd666b"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.969175 4837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 19:27:17.972594 4852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:17.974020 4837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:17.974146 4837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "1b1c017649a3428a8d0a5c48663daf7d"
format_stamp: "Formatted at 2026-09-26 19:27:17 on dist-test-slave-9wb0"
server_key: "b6b399a51c9eefaac713ad11d4dc9356"
server_key_iv: "544d3a4accdba71b9ae43914cfdd666b"
server_key_version: "encryptionkey@0"
I20260926 19:27:17.974303 4837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:17.998054 4837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:18.002190 4837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:18.002439 4837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:18.003346 4837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:18.004593 4837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:18.004664 4837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.004755 4837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:18.004797 4837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.037990 4837 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:42423
I20260926 19:27:18.038089 4964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:42423 every 8 connection(s)
I20260926 19:27:18.039273 4837 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:18.041931 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4837
I20260926 19:27:18.042181 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:18.043215 1180 external_mini_cluster.cc:1428] Setting key 9c99b38f36b4c580ed39873bfef6b97c
Sep 26 19:27:18 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450837, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:18.059278 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:18.041854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:35171 (local address 127.1.39.62:38561)
0926 19:27:18.043974 (+ 2120us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:18.043981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:18.044003 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:27:18.045548 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:18.045553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:18.045641 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:18.045766 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:18.047217 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.047917 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.049488 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.049895 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.050032 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:18.053719 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:18.053730 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:18.053733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:18.053763 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:18.056289 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:18.057011 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:18.057023 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:18.057026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:18.057143 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:18.057606 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:18.057611 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:18.057615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:18.057869 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:18.058038 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:18.058703 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 19:27:18.058914 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":1898,"threads_started":1}
I20260926 19:27:18.060257 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:18.042408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38561 (local address 127.1.39.1:35171)
0926 19:27:18.043269 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:18.043319 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:18.045243 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:18.046001 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:18.046009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:18.046499 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:18.047054 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:18.047073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.048067 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.048073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:18.049347 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:18.049365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.050128 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.050149 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:18.050392 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:18.051094 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:18.051117 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:18.053494 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:18.056436 (+ 2942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:18.056445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:18.056464 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:18.056883 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:18.057260 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:18.057265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:18.057269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:18.057480 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:18.058006 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:18.058014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:18.058345 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 19:27:18.058614 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 19:27:18.059133 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:27:18.061925 4965 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38561
I20260926 19:27:18.062258 4965 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:18.062963 4965 heartbeater.cc:511] Master 127.1.39.62:38561 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
I20260926 19:27:18.065063 4776 ts_manager.cc:194] Registered new tserver with Master: 1b1c017649a3428a8d0a5c48663daf7d (127.1.39.1:42423)
I20260926 19:27:18.066520 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:35171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:18.086740 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:18.406163 4971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:18.406436 4971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:18.406603 4971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:18.413640 4971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:18.413738 4971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:18.413944 4971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:18.423285 4971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:18.424541 4971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:18.425892 4971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:18.439313 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:18.439281 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:18.439899 4971 server_base.cc:1034] running on GCE node
W20260926 19:27:18.441515 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:18.442211 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:18.443104 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:18.444329 4971 hybrid_clock.cc:648] HybridClock initialized: now 1790450838444283 us; error 39 us; skew 500 ppm
I20260926 19:27:18.444645 4971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:18 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:18.447721 4971 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:18.449182 4971 webserver.cc:460] Webserver started at http://127.1.39.2:45943/ using document root <none> and password file <none>
I20260926 19:27:18.449856 4971 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:18.449978 4971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:18.450294 4971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:18.453058 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "6490e38ffe444118bd50ac543313de90"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "63d1654e51c8da4d42f5e0f9b2d9dfe1"
server_key_iv: "bf115e2d52762190ffaed9b0b57dd148"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.453886 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "6490e38ffe444118bd50ac543313de90"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "63d1654e51c8da4d42f5e0f9b2d9dfe1"
server_key_iv: "bf115e2d52762190ffaed9b0b57dd148"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.458791 4971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:27:18.462194 4986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.463487 4971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:27:18.463667 4971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "6490e38ffe444118bd50ac543313de90"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "63d1654e51c8da4d42f5e0f9b2d9dfe1"
server_key_iv: "bf115e2d52762190ffaed9b0b57dd148"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.463862 4971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:18.495548 4971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:18.501024 4971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:18.501377 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:18.502586 4971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:18.504240 4971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:18.504314 4971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.504396 4971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:18.504448 4971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.536362 4971 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:35537
I20260926 19:27:18.536410 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:35537 every 8 connection(s)
I20260926 19:27:18.537698 4971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:18.545646 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 4971
I20260926 19:27:18.545859 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:18.546412 1180 external_mini_cluster.cc:1428] Setting key 49fb4f647be2f06768dfcad398f3f5cb
Sep 26 19:27:18 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:18.558780 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:18.540122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:46025 (local address 127.1.39.62:38561)
0926 19:27:18.540286 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:18.540291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:18.541541 (+ 1250us) server_negotiation.cc:407] Connection header received
0926 19:27:18.543325 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:18.543329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:18.543393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:18.543486 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:18.544874 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.545617 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.548163 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.548653 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.548787 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:18.552360 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:18.552365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:18.552368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:18.552398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:18.555565 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:18.556252 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:18.556261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:18.556267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:18.556420 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:18.556834 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:18.556841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:18.556847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:18.557217 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:18.557402 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:18.558092 (+ 690us) server_negotiation.cc:299] Negotiation successful
0926 19:27:18.558343 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:27:18.559435 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:18.540556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38561 (local address 127.1.39.2:46025)
0926 19:27:18.541381 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:18.541434 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:18.543087 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:18.543687 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:18.543695 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:18.544193 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:18.544732 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:18.544749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.545774 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.545780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:18.547987 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:18.548000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:18.548877 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:18.548885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:18.549114 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:18.549846 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:18.549867 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:18.552148 (+ 2281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:18.555711 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:18.555717 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:18.555730 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:18.556137 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:18.556533 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:18.556537 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:18.556539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:18.556726 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:18.557398 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:18.557403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:18.557735 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 19:27:18.557957 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 19:27:18.558324 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
I20260926 19:27:18.560904 5099 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38561
I20260926 19:27:18.561264 5099 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:18.562175 5099 heartbeater.cc:511] Master 127.1.39.62:38561 requested a full tablet report, sending...
I20260926 19:27:18.563766 4776 ts_manager.cc:194] Registered new tserver with Master: 6490e38ffe444118bd50ac543313de90 (127.1.39.2:35537)
I20260926 19:27:18.564576 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:46025
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:18.596840 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:18.920553 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:18.920903 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:18.921078 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:18.928679 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:18.928846 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:18.929051 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:18.939118 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:18.940454 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:18.941892 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:18.955963 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:18.956049 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:18.956302 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:18.957192 5104 server_base.cc:1034] running on GCE node
I20260926 19:27:18.957876 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:18.958825 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:18.960052 5104 hybrid_clock.cc:648] HybridClock initialized: now 1790450838959999 us; error 46 us; skew 500 ppm
I20260926 19:27:18.960398 5104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:18 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:18.963868 5104 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:18.965564 5104 webserver.cc:460] Webserver started at http://127.1.39.3:35067/ using document root <none> and password file <none>
I20260926 19:27:18.966235 5104 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:18.966336 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:18.966648 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:18.969726 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "b6c2e2f60245495f8bd59981a749bd2c"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "f55c8b2e5070314733b77aa7b4309b66"
server_key_iv: "56d51bcc1a82678474006e1fdf082c08"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.970440 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "b6c2e2f60245495f8bd59981a749bd2c"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "f55c8b2e5070314733b77aa7b4309b66"
server_key_iv: "56d51bcc1a82678474006e1fdf082c08"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.975276 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:27:18.978763 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:18.980054 5104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:27:18.980222 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "b6c2e2f60245495f8bd59981a749bd2c"
format_stamp: "Formatted at 2026-09-26 19:27:18 on dist-test-slave-9wb0"
server_key: "f55c8b2e5070314733b77aa7b4309b66"
server_key_iv: "56d51bcc1a82678474006e1fdf082c08"
server_key_version: "encryptionkey@0"
I20260926 19:27:18.980422 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:19.017146 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:19.021657 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:19.021970 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:19.022914 5104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:19.024240 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:19.024343 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:19.024499 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:19.024576 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:19.058115 5104 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38927
I20260926 19:27:19.058202 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38927 every 8 connection(s)
I20260926 19:27:19.059394 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:19.065132 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5104
I20260926 19:27:19.065411 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:19.065943 1180 external_mini_cluster.cc:1428] Setting key df76a1047a5a1b6d199d508d9e1ab14c
I20260926 19:27:19.069995 4965 heartbeater.cc:503] Master 127.1.39.62:38561 was elected leader, sending a full tablet report...
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450838, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:19.080281 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.062016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:57401 (local address 127.1.39.62:38561)
0926 19:27:19.062479 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.062486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.063531 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 19:27:19.065343 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.065351 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.065458 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.065595 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.067257 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.068406 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.070090 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.070584 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.070714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.074578 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.074583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.074587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.074618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.077174 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.078047 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.078054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.078060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.078180 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.078641 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.078644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.078646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.078909 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.079013 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.079725 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.079912 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":204,"threads_started":1}
I20260926 19:27:19.081622 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.062694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38561 (local address 127.1.39.3:57401)
0926 19:27:19.063374 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:19.063410 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:19.065039 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:19.065780 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:19.065789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:19.066347 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:19.067059 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:19.067078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.068554 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.068558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:19.069944 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:19.069955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.070855 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.070863 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:19.071176 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.072026 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:19.072057 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:19.074351 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:19.077341 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:19.077350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:19.077370 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:19.077916 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:19.078278 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:19.078282 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:19.078284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:19.078537 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:19.079015 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:19.079021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:19.079359 (+ 338us) client_negotiation.cc:769] Sending connection context
0926 19:27:19.079606 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 19:27:19.080163 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":291,"threads_started":1}
I20260926 19:27:19.082917 5232 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38561
I20260926 19:27:19.083227 5232 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:19.083992 5232 heartbeater.cc:511] Master 127.1.39.62:38561 requested a full tablet report, sending...
I20260926 19:27:19.085585 4776 ts_manager.cc:194] Registered new tserver with Master: b6c2e2f60245495f8bd59981a749bd2c (127.1.39.3:38927)
I20260926 19:27:19.086504 4776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:57401
I20260926 19:27:19.093998 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:19.140740 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.126691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.1.39.1:42423)
0926 19:27:19.127351 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.127358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.127384 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:27:19.127441 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.127447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.127649 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.127786 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.129013 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.130241 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.131448 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.132016 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.132112 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.135836 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.135846 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.135859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.135896 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.138528 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.139110 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.139116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.139118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.139236 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.139557 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.139560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.139562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.139859 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.140082 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.140290 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.140468 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":347,"threads_started":1}
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:19.157681 5234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.144882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.1.39.62:38561)
0926 19:27:19.145101 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.145107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.145396 (+ 289us) server_negotiation.cc:407] Connection header received
0926 19:27:19.145551 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.145556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.145667 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.145762 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.147096 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.147822 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.148983 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.149617 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.149737 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.153247 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.153252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.153255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.153283 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.155544 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.156078 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.156082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.156084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.156187 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.156473 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.156476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.156478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.156740 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.156903 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.157146 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.157351 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 19:27:19.158239 4776 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45166
I20260926 19:27:19.159185 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4837
I20260926 19:27:19.171579 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.159992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55146 (local address 127.1.39.1:42423)
0926 19:27:19.160217 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.160225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.160457 (+ 232us) server_negotiation.cc:407] Connection header received
0926 19:27:19.160650 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.160655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.160754 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.160859 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.162061 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.162736 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.163911 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.164316 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.164446 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.166671 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.166677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.166681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.166717 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.169183 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.169819 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.169826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.169834 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.169943 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.170319 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.170324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.170327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.170557 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.170787 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.171082 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.171295 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:27:19.172127 4960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:19.255527 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4837
I20260926 19:27:19.271836 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4971
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:19.288085 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.272810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.1.39.2:35537)
0926 19:27:19.273410 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.273417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.273444 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:27:19.273500 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.273506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.273722 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.273852 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.274946 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.276014 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.277248 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.277899 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.277998 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.281926 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.281937 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.281952 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.281983 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.284992 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.285842 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.285848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.285851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.285971 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.286411 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.286418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.286422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.286745 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.287036 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.287403 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.287657 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":296,"threads_started":1}
I20260926 19:27:19.289528 5094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:19.372123 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4971
I20260926 19:27:19.388588 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5104
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[4731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:19.405391 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:19.389757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.1.39.3:38927)
0926 19:27:19.390353 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:19.390360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:19.390389 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:27:19.390446 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:19.390452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:19.390670 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:19.390842 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:19.392267 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.393504 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.394815 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:19.395404 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:19.395543 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:19.399822 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:19.399832 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:19.399846 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:19.399883 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:19.402646 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.403222 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.403229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.403231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.403346 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:19.403704 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:19.403708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:19.403710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:19.404060 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:19.404314 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:19.404735 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 19:27:19.405033 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":285,"threads_started":1}
I20260926 19:27:19.406693 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:19.488422 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5104
I20260926 19:27:19.504801 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 4744
I20260926 19:27:19.506021 4806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:19.607609 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 4744
2026-09-26T19:27:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3866 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[5252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[5252](info): set up 2 sockets
Sep 26 19:27:19 dist-test-slave-9wb0 krb5kdc[5252](info): commencing operation
krb5kdc: starting...
W20260926 19:27:20.710034 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:20 dist-test-slave-9wb0 krb5kdc[5252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:20.908841 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33773
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:21.230389 5265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:21.230676 5265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:21.230886 5265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:21.238176 5265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:21.238287 5265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:21.238379 5265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:21.238474 5265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:21.238548 5265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:21.248322 5265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:21.249678 5265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:21.251046 5265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:21.262267 5270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:21.262310 5271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:21.262337 5273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:21.263552 5265 server_base.cc:1034] running on GCE node
I20260926 19:27:21.264309 5265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:21.265753 5265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:21.266983 5265 hybrid_clock.cc:648] HybridClock initialized: now 1790450841266947 us; error 53 us; skew 500 ppm
I20260926 19:27:21.267328 5265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:21 dist-test-slave-9wb0 krb5kdc[5252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:21.270403 5265 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:21.271637 5265 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:21.271770 5265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:21.272085 5265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:21.275050 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "bc76d86070df4cbb88b0001548a070a3"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "be797ccd36674dab20ab8e659cae79f8"
server_key_iv: "44671951c4211558ffece57c6338e16e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.275844 5265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "bc76d86070df4cbb88b0001548a070a3"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "be797ccd36674dab20ab8e659cae79f8"
server_key_iv: "44671951c4211558ffece57c6338e16e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.280953 5265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:27:21.284343 5279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:21.285650 5265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:21.285830 5265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "bc76d86070df4cbb88b0001548a070a3"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "be797ccd36674dab20ab8e659cae79f8"
server_key_iv: "44671951c4211558ffece57c6338e16e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.286048 5265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:21.312141 5265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:21.316351 5265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:21.316668 5265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:21.335536 5265 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33773
I20260926 19:27:21.335562 5330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33773 every 8 connection(s)
I20260926 19:27:21.336467 5265 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:21.338095 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5265
I20260926 19:27:21.338321 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:21.339116 1180 external_mini_cluster.cc:1428] Setting key 945356e71c4d67810a81a44fb68453d2
I20260926 19:27:21.341588 5331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:27:21 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:21.354677 5331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: Bootstrap starting.
I20260926 19:27:21.357889 5331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:21.358891 5331 log.cc:824] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:21.361608 5331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: No bootstrap required, opened a new log
I20260926 19:27:21.362512 5334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:21.341183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51362 (local address 127.1.39.62:33773)
0926 19:27:21.342219 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:21.342231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:21.342272 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:27:21.343910 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:21.343932 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:21.344324 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:21.344654 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:21.346536 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.348132 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.349872 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.350597 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.350735 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:21.355435 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:21.355442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:21.355464 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:21.355501 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:21.358583 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:21.359319 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:21.359325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:21.359331 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:21.359516 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:21.359901 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:21.359905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:21.359907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:21.360389 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:21.360685 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:21.361123 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 19:27:21.361534 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":831,"thread_start_us":478,"threads_started":1}
I20260926 19:27:21.368927 5331 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } }
I20260926 19:27:21.369338 5331 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:21.369459 5331 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc76d86070df4cbb88b0001548a070a3, State: Initialized, Role: FOLLOWER
I20260926 19:27:21.369997 5331 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } }
I20260926 19:27:21.370257 5331 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:21.370401 5331 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:21.370591 5331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:21.372910 5331 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } }
I20260926 19:27:21.373412 5331 leader_election.cc:302] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc76d86070df4cbb88b0001548a070a3; no voters:
I20260926 19:27:21.373956 5331 leader_election.cc:288] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:21.374076 5336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:21.374765 5336 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [term 1 LEADER]: Becoming Leader. State: Replica: bc76d86070df4cbb88b0001548a070a3, State: Running, Role: LEADER
I20260926 19:27:21.375241 5336 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } }
I20260926 19:27:21.376086 5331 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:21.378062 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc76d86070df4cbb88b0001548a070a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } } }
I20260926 19:27:21.378021 5338 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc76d86070df4cbb88b0001548a070a3. Latest consensus state: current_term: 1 leader_uuid: "bc76d86070df4cbb88b0001548a070a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc76d86070df4cbb88b0001548a070a3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33773 } } }
I20260926 19:27:21.378396 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:21.378400 5338 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:21.379444 5343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:21.385761 5343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:21.394150 5343 catalog_manager.cc:1347] Generated new cluster ID: 02d0a655a00f428d898eccf3fa120852
I20260926 19:27:21.394268 5343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:21.402825 5343 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:21.404038 5343 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:21.412926 5343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: Generated new TSK 0
I20260926 19:27:21.413770 5343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:21.461153 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:21.790762 5357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:21.791131 5357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:21.791245 5357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:21.798755 5357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:21.798863 5357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:21.799006 5357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:21.808845 5357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:21.810117 5357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:21.811424 5357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:21.824651 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:21.824733 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:21.824712 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:21.826040 5357 server_base.cc:1034] running on GCE node
I20260926 19:27:21.826714 5357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:21.827667 5357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:21.828892 5357 hybrid_clock.cc:648] HybridClock initialized: now 1790450841828835 us; error 54 us; skew 500 ppm
I20260926 19:27:21.829218 5357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:21 dist-test-slave-9wb0 krb5kdc[5252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:21.832261 5357 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:21.833529 5357 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:21.833664 5357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:21.833974 5357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:21.836788 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "fbc8eccc7ae046b381dfb481007d49ea"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "9b9c8b067570ad8bdb08602b2a46cd0a"
server_key_iv: "a364e7ddd4cbd33a6bafa548fcc7d24e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.837666 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "fbc8eccc7ae046b381dfb481007d49ea"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "9b9c8b067570ad8bdb08602b2a46cd0a"
server_key_iv: "a364e7ddd4cbd33a6bafa548fcc7d24e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.842504 5357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:27:21.845821 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:21.847026 5357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:27:21.847208 5357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "fbc8eccc7ae046b381dfb481007d49ea"
format_stamp: "Formatted at 2026-09-26 19:27:21 on dist-test-slave-9wb0"
server_key: "9b9c8b067570ad8bdb08602b2a46cd0a"
server_key_iv: "a364e7ddd4cbd33a6bafa548fcc7d24e"
server_key_version: "encryptionkey@0"
I20260926 19:27:21.847419 5357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:21.864235 5357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:21.868494 5357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:21.868702 5357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:21.869566 5357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:21.870821 5357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:21.870910 5357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:21.870997 5357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:21.871052 5357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:21.902117 5357 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:44741
I20260926 19:27:21.902168 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:44741 every 8 connection(s)
I20260926 19:27:21.902971 5357 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:21.910317 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5357
I20260926 19:27:21.910549 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:21.911155 1180 external_mini_cluster.cc:1428] Setting key b1b6a12c5f5a87a1f1224a01006ce720
Sep 26 19:27:21 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450841, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:21.923775 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:21.905450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:33243 (local address 127.1.39.62:33773)
0926 19:27:21.905903 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:21.905907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:21.906794 (+ 887us) server_negotiation.cc:407] Connection header received
0926 19:27:21.908560 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:21.908566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:21.908651 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:21.908767 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:21.910519 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.911321 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.913186 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.913671 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.913769 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:21.917906 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:21.917914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:21.917922 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:21.917965 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:21.920843 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:21.921585 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:21.921591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:21.921597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:21.921720 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:21.922132 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:21.922137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:21.922140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:21.922403 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:21.922587 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:21.923222 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 19:27:21.923430 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":251,"threads_started":1}
I20260926 19:27:21.924690 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:21.905922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33773 (local address 127.1.39.1:33243)
0926 19:27:21.906641 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:21.906684 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:21.908279 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:21.908986 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:21.908994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:21.909562 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:21.910316 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:21.910347 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.911475 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.911481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:21.913036 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:21.913050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:21.913893 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:21.913900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:21.914211 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:21.915011 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:21.915045 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:21.917674 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:21.920959 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:21.920965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:21.920977 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:21.921436 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:21.921830 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:21.921837 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:21.921840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:21.922008 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:21.922560 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:21.922575 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:21.922882 (+ 307us) client_negotiation.cc:769] Sending connection context
0926 19:27:21.923134 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 19:27:21.923645 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":297,"threads_started":1}
I20260926 19:27:21.926249 5484 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33773
I20260926 19:27:21.926558 5484 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:21.927192 5484 heartbeater.cc:511] Master 127.1.39.62:33773 requested a full tablet report, sending...
I20260926 19:27:21.929493 5296 ts_manager.cc:194] Registered new tserver with Master: fbc8eccc7ae046b381dfb481007d49ea (127.1.39.1:44741)
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:27:21.931641 5296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:33243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:21.956336 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:22.286686 5490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:22.286965 5490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:22.287127 5490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:22.294533 5490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:22.294643 5490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:22.294848 5490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:22.305160 5490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:22.306516 5490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:22.307854 5490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:22.320902 5495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:22.320976 5498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:22.320966 5496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:22.322438 5490 server_base.cc:1034] running on GCE node
I20260926 19:27:22.322971 5490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:22.323872 5490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:22.325124 5490 hybrid_clock.cc:648] HybridClock initialized: now 1790450842325084 us; error 35 us; skew 500 ppm
I20260926 19:27:22.325490 5490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:22 dist-test-slave-9wb0 krb5kdc[5252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:22.328558 5490 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:22.329819 5490 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:22.329954 5490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:22.330372 5490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:22.333249 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "bbc1914e68bd4bac8069131e1a5234d9"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "5e3cea41375cfd6897562a61959992e7"
server_key_iv: "516def06861e54539aece78281d1e6ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.334074 5490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc1914e68bd4bac8069131e1a5234d9"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "5e3cea41375cfd6897562a61959992e7"
server_key_iv: "516def06861e54539aece78281d1e6ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.340351 5490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 19:27:22.344659 5504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.346185 5490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:22.346360 5490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "bbc1914e68bd4bac8069131e1a5234d9"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "5e3cea41375cfd6897562a61959992e7"
server_key_iv: "516def06861e54539aece78281d1e6ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.346585 5490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:22.368635 5490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:22.372946 5490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:22.373234 5490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:22.374189 5490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:22.375509 5490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:22.375631 5490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.375765 5490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:22.375854 5490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.407568 5490 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:42307
I20260926 19:27:22.407655 5616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:42307 every 8 connection(s)
I20260926 19:27:22.408406 5490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:22.415880 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5490
I20260926 19:27:22.416097 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:22.416541 1180 external_mini_cluster.cc:1428] Setting key 7416c06b1d76d742bd7c004bbfb3b8cd
Sep 26 19:27:22 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:22.428872 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.410817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:39657 (local address 127.1.39.62:33773)
0926 19:27:22.410992 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.410996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.412189 (+ 1193us) server_negotiation.cc:407] Connection header received
0926 19:27:22.414035 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.414040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.414128 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.414246 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:22.415875 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.416696 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.418709 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.419106 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.419251 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:22.423181 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:22.423186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:22.423189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:22.423219 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:22.425985 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.426682 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.426690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.426699 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.426814 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.427231 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.427236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.427239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.427483 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:22.427638 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:22.428316 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 19:27:22.428531 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:27:22.429935 5619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.411306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33773 (local address 127.1.39.2:39657)
0926 19:27:22.412026 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:22.412070 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:22.413772 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:22.414493 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:22.414502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:22.415027 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:22.415691 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.415717 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.416845 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.416852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:22.418577 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.418591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.419340 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.419350 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:22.419644 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:22.420363 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:22.420404 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:22.422953 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:22.426119 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:22.426128 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:22.426144 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:22.426554 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:22.426923 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:22.426928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:22.426932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:22.427112 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:22.427622 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:22.427630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:22.427950 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 19:27:22.428228 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 19:27:22.428740 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":305,"threads_started":1}
I20260926 19:27:22.431444 5617 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33773
I20260926 19:27:22.431773 5617 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:22.432411 5617 heartbeater.cc:511] Master 127.1.39.62:33773 requested a full tablet report, sending...
I20260926 19:27:22.433902 5296 ts_manager.cc:194] Registered new tserver with Master: bbc1914e68bd4bac8069131e1a5234d9 (127.1.39.2:42307)
I20260926 19:27:22.435127 5296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:39657
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:22.462915 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:38065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:22.784045 5622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:22.784327 5622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:22.784423 5622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:22.791599 5622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:22.791711 5622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:22.791831 5622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:22.801568 5622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:38065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:22.802901 5622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:22.804128 5622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:22.816742 5627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:22.816777 5628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:22.816840 5630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:22.817858 5622 server_base.cc:1034] running on GCE node
I20260926 19:27:22.818455 5622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:22.819378 5622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:22.820583 5622 hybrid_clock.cc:648] HybridClock initialized: now 1790450842820533 us; error 39 us; skew 500 ppm
I20260926 19:27:22.820909 5622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:22 dist-test-slave-9wb0 krb5kdc[5252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:22.824229 5622 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:22.825464 5622 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:22.825613 5622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:22.825937 5622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:22.828755 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "8b1f4277056c48bab01fc9555065917a"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "29cc1c0176d774e3f7ba74fba15d1258"
server_key_iv: "b61e648bdd3e2e5448d59794b0023b8f"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.829607 5622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "8b1f4277056c48bab01fc9555065917a"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "29cc1c0176d774e3f7ba74fba15d1258"
server_key_iv: "b61e648bdd3e2e5448d59794b0023b8f"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.834583 5622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:27:22.837939 5636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.839152 5622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:22.839334 5622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "8b1f4277056c48bab01fc9555065917a"
format_stamp: "Formatted at 2026-09-26 19:27:22 on dist-test-slave-9wb0"
server_key: "29cc1c0176d774e3f7ba74fba15d1258"
server_key_iv: "b61e648bdd3e2e5448d59794b0023b8f"
server_key_version: "encryptionkey@0"
I20260926 19:27:22.839550 5622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:22.865065 5622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:22.869362 5622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:22.869637 5622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:22.870615 5622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:22.871917 5622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:22.872051 5622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.872180 5622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:22.872268 5622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:22.905025 5622 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38113
I20260926 19:27:22.905043 5748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38113 every 8 connection(s)
I20260926 19:27:22.905947 5622 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:22.912149 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5622
I20260926 19:27:22.912360 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:22.912801 1180 external_mini_cluster.cc:1428] Setting key 03e6362b5cfd5ec9dd905ed18b773872
Sep 26 19:27:22 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450842, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:22.926311 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.908596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:59167 (local address 127.1.39.62:33773)
0926 19:27:22.908790 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.908794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.909965 (+ 1171us) server_negotiation.cc:407] Connection header received
0926 19:27:22.911929 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.911937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.912031 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.912157 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:22.913966 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.914914 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.916835 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.917352 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.917492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:22.920928 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:22.920933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:22.920936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:22.920971 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:22.923505 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.924153 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.924157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.924159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.924278 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.924727 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.924730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.924732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.925037 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:22.925158 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:22.925877 (+ 719us) server_negotiation.cc:299] Negotiation successful
0926 19:27:22.926057 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:27:22.927520 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.908890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33773 (local address 127.1.39.3:59167)
0926 19:27:22.909739 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:22.909780 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:22.911618 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:22.912527 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:22.912535 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:22.913125 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:22.913787 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.913806 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.915097 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.915101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:22.916678 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.916690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.917578 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.917583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:22.917763 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:22.918452 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:22.918482 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:22.920709 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:22.923637 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:22.923643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:22.923656 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:22.924041 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:22.924408 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:22.924412 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:22.924414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:22.924618 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:22.925152 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:22.925157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:22.925528 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 19:27:22.925780 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 19:27:22.926252 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":402,"threads_started":1}
I20260926 19:27:22.929069 5749 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33773
I20260926 19:27:22.929417 5749 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:22.930047 5749 heartbeater.cc:511] Master 127.1.39.62:33773 requested a full tablet report, sending...
I20260926 19:27:22.931515 5296 ts_manager.cc:194] Registered new tserver with Master: 8b1f4277056c48bab01fc9555065917a (127.1.39.3:38113)
I20260926 19:27:22.932360 5296 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:59167
I20260926 19:27:22.935119 5484 heartbeater.cc:503] Master 127.1.39.62:33773 was elected leader, sending a full tablet report...
I20260926 19:27:22.940618 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:27:22.957549 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.945078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.1.39.62:33773)
0926 19:27:22.945262 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.945267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.945800 (+ 533us) server_negotiation.cc:407] Connection header received
0926 19:27:22.945929 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.945933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.946008 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.946092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:22.947447 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.948218 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.949606 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.950277 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.950436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:22.952711 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:22.952716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:22.952719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:22.952749 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:22.955322 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.955924 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.955928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.955931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.956054 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:22.956348 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:22.956351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:22.956353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:22.956658 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:22.956833 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:22.957100 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 19:27:22.957252 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:27:22.961694 5296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:22.968772 5296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:23.003940 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.985376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.1.39.2:42307)
0926 19:27:22.995291 (+ 9915us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.995307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.995350 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:27:22.995516 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.995529 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.995896 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.996224 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:22.996994 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.998774 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.000802 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.002671 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.002808 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.003043 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.003405 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.003602 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9675,"thread_start_us":9531,"threads_started":1}
I20260926 19:27:23.003969 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.985100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42307 (local address 127.0.0.1:57736)
0926 19:27:22.994516 (+ 9416us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:22.994538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:22.994658 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:22.996244 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:22.996257 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:22.996304 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:22.996781 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.996793 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:22.998931 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:22.998939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.000641 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.000653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.002884 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.002890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.003122 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.003179 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:27:23.003327 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 19:27:23.003557 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9233,"thread_start_us":9053,"threads_started":1}
I20260926 19:27:23.006050 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.981888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.1.39.1:44741)
0926 19:27:22.997115 (+ 15227us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.997124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.997155 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:27:22.997588 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.997600 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.997851 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.998104 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:22.999015 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.000802 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.003501 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.004903 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.005060 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.005309 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.005561 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.005717 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14990,"thread_start_us":14763,"threads_started":1}
I20260926 19:27:23.006086 5762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.981284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:44741 (local address 127.0.0.1:55268)
0926 19:27:22.993712 (+ 12428us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:22.993741 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:22.993908 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:22.998431 (+ 4523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:22.998439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:22.998481 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:22.998835 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.998874 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.000959 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.000965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.003265 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.003277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.005110 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.005117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.005325 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.005365 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:27:23.005515 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 19:27:23.005661 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12226,"mutex_wait_us":67,"thread_start_us":758,"threads_started":1}
I20260926 19:27:23.008216 5761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.981242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38113 (local address 127.0.0.1:40602)
0926 19:27:22.982022 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:22.982068 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:22.982237 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:22.998098 (+ 15861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:22.998103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:22.998132 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:22.998556 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:22.998567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.001066 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.001076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.003530 (+ 2454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.003543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.007432 (+ 3889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.007443 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.007629 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.007662 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:27:23.007734 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 19:27:23.007864 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":23,"thread_start_us":12045,"threads_started":1}
I20260926 19:27:23.008675 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 (DEFAULT_TABLE table=test-table [id=c00166050f8445edbf65e52dc07a3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:23.009399 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:22.981392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.1.39.3:38113)
0926 19:27:22.997481 (+ 16089us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:22.997490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:22.997521 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:27:22.997615 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:22.997623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:22.997869 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:22.998782 (+ 913us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:22.999134 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.000873 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.003716 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.005256 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.008561 (+ 3305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.008801 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.008872 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.009042 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15874,"thread_start_us":15658,"threads_started":1}
I20260926 19:27:23.010377 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ece7c276d04ccfba9572aa00a3d424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:23.011996 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 (DEFAULT_TABLE table=test-table [id=c00166050f8445edbf65e52dc07a3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:23.012951 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 (DEFAULT_TABLE table=test-table [id=c00166050f8445edbf65e52dc07a3c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:23.013726 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ece7c276d04ccfba9572aa00a3d424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:23.014297 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ece7c276d04ccfba9572aa00a3d424. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:23.022647 5767 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Bootstrap starting.
I20260926 19:27:23.025939 5767 tablet_bootstrap.cc:654] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:23.026962 5767 log.cc:824] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:23.028194 5768 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Bootstrap starting.
I20260926 19:27:23.029053 5769 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Bootstrap starting.
I20260926 19:27:23.031980 5768 tablet_bootstrap.cc:654] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:23.033172 5768 log.cc:824] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:23.033229 5769 tablet_bootstrap.cc:654] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:23.034540 5769 log.cc:824] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:23.034951 5767 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: No bootstrap required, opened a new log
I20260926 19:27:23.035291 5767 ts_tablet_manager.cc:1397] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260926 19:27:23.035840 5768 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: No bootstrap required, opened a new log
I20260926 19:27:23.036094 5768 ts_tablet_manager.cc:1397] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260926 19:27:23.037536 5769 tablet_bootstrap.cc:492] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: No bootstrap required, opened a new log
I20260926 19:27:23.037822 5769 ts_tablet_manager.cc:1397] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:27:23.044025 5768 raft_consensus.cc:348] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.044433 5768 raft_consensus.cc:374] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:23.044543 5768 raft_consensus.cc:729] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc8eccc7ae046b381dfb481007d49ea, State: Initialized, Role: FOLLOWER
I20260926 19:27:23.045166 5768 consensus_queue.cc:260] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.046783 5768 ts_tablet_manager.cc:1428] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260926 19:27:23.046904 5767 raft_consensus.cc:348] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.047348 5767 raft_consensus.cc:374] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:23.047470 5767 raft_consensus.cc:729] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc1914e68bd4bac8069131e1a5234d9, State: Initialized, Role: FOLLOWER
I20260926 19:27:23.048149 5767 consensus_queue.cc:260] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.049340 5617 heartbeater.cc:503] Master 127.1.39.62:33773 was elected leader, sending a full tablet report...
I20260926 19:27:23.049966 5767 ts_tablet_manager.cc:1428] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260926 19:27:23.049845 5769 raft_consensus.cc:348] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.050304 5769 raft_consensus.cc:374] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:23.050432 5769 raft_consensus.cc:729] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1f4277056c48bab01fc9555065917a, State: Initialized, Role: FOLLOWER
I20260926 19:27:23.051054 5769 consensus_queue.cc:260] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.053607 5769 ts_tablet_manager.cc:1428] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260926 19:27:23.053752 5749 heartbeater.cc:503] Master 127.1.39.62:33773 was elected leader, sending a full tablet report...
W20260926 19:27:23.155169 5485 tablet.cc:2369] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:27:23.157533 5750 tablet.cc:2369] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:27:23.160229 5618 tablet.cc:2369] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:23.162652 5775 raft_consensus.cc:482] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:23.162917 5775 raft_consensus.cc:504] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.164598 5775 leader_election.cc:288] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbc8eccc7ae046b381dfb481007d49ea (127.1.39.1:44741), bbc1914e68bd4bac8069131e1a5234d9 (127.1.39.2:42307)
I20260926 19:27:23.171137 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.164923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42307 (local address 127.1.39.3:57015)
0926 19:27:23.165269 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:23.165323 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:23.165522 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:23.165957 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:23.165961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:23.166013 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:23.166245 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.166257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.167484 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.167490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.169060 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.169072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.170347 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.170353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.170579 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.170641 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 19:27:23.170734 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 19:27:23.170896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260926 19:27:23.171165 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.165028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:57015 (local address 127.1.39.2:42307)
0926 19:27:23.165255 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.165263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.165423 (+ 160us) server_negotiation.cc:407] Connection header received
0926 19:27:23.165718 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.165723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.165805 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.165919 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:23.166395 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.167340 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.169222 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.170175 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.170279 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.170345 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.170770 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.170894 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 19:27:23.171546 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.165096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:33483 (local address 127.1.39.1:44741)
0926 19:27:23.165266 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.165272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.166019 (+ 747us) server_negotiation.cc:407] Connection header received
0926 19:27:23.166184 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.166193 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.166271 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.166366 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:23.166963 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.167857 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.169674 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.170583 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.170739 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.170833 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.171176 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.171345 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:27:23.171571 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.165069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:44741 (local address 127.1.39.3:33483)
0926 19:27:23.165910 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:23.165927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:23.166044 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:23.166389 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:23.166393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:23.166415 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:23.166798 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.166809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.168016 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.168019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.169509 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:23.169522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.170826 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.170832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:23.170993 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.171033 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:27:23.171130 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 19:27:23.171260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":1,"thread_start_us":496,"threads_started":1}
I20260926 19:27:23.172647 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ece7c276d04ccfba9572aa00a3d424" candidate_uuid: "8b1f4277056c48bab01fc9555065917a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc8eccc7ae046b381dfb481007d49ea" is_pre_election: true
I20260926 19:27:23.172621 5572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ece7c276d04ccfba9572aa00a3d424" candidate_uuid: "8b1f4277056c48bab01fc9555065917a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc1914e68bd4bac8069131e1a5234d9" is_pre_election: true
I20260926 19:27:23.173108 5572 raft_consensus.cc:2454] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1f4277056c48bab01fc9555065917a in term 0.
I20260926 19:27:23.173130 5439 raft_consensus.cc:2454] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b1f4277056c48bab01fc9555065917a in term 0.
I20260926 19:27:23.173776 5638 leader_election.cc:302] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b1f4277056c48bab01fc9555065917a, bbc1914e68bd4bac8069131e1a5234d9; no voters:
I20260926 19:27:23.174115 5775 raft_consensus.cc:2781] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:23.174269 5775 raft_consensus.cc:482] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:23.174376 5775 raft_consensus.cc:3037] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:23.176532 5775 raft_consensus.cc:504] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.177179 5775 leader_election.cc:288] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [CANDIDATE]: Term 1 election: Requested vote from peers fbc8eccc7ae046b381dfb481007d49ea (127.1.39.1:44741), bbc1914e68bd4bac8069131e1a5234d9 (127.1.39.2:42307)
I20260926 19:27:23.177646 5572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ece7c276d04ccfba9572aa00a3d424" candidate_uuid: "8b1f4277056c48bab01fc9555065917a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc1914e68bd4bac8069131e1a5234d9"
I20260926 19:27:23.177711 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ece7c276d04ccfba9572aa00a3d424" candidate_uuid: "8b1f4277056c48bab01fc9555065917a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc8eccc7ae046b381dfb481007d49ea"
I20260926 19:27:23.177814 5572 raft_consensus.cc:3037] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:23.177959 5439 raft_consensus.cc:3037] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:23.180408 5572 raft_consensus.cc:2454] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1f4277056c48bab01fc9555065917a in term 1.
I20260926 19:27:23.180701 5439 raft_consensus.cc:2454] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b1f4277056c48bab01fc9555065917a in term 1.
I20260926 19:27:23.180787 5638 leader_election.cc:302] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b1f4277056c48bab01fc9555065917a, bbc1914e68bd4bac8069131e1a5234d9; no voters:
I20260926 19:27:23.181070 5775 raft_consensus.cc:2781] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:23.181571 5775 raft_consensus.cc:686] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 1 LEADER]: Becoming Leader. State: Replica: 8b1f4277056c48bab01fc9555065917a, State: Running, Role: LEADER
I20260926 19:27:23.182106 5775 consensus_queue.cc:237] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } }
I20260926 19:27:23.186172 5295 catalog_manager.cc:5495] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b1f4277056c48bab01fc9555065917a (127.1.39.3). New cstate: current_term: 1 leader_uuid: "8b1f4277056c48bab01fc9555065917a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b1f4277056c48bab01fc9555065917a" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:23.238241 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.233283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.1.39.3:38113)
0926 19:27:23.233489 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.233494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.233621 (+ 127us) server_negotiation.cc:407] Connection header received
0926 19:27:23.233790 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.233794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.233874 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.233999 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:23.234373 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.235044 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.236284 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.236657 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.236742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.237166 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:23.237725 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:23.237831 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.237969 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.238030 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:27:23.246794 5439 raft_consensus.cc:1261] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 1 FOLLOWER]: Refusing update from remote peer 8b1f4277056c48bab01fc9555065917a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:23.246794 5572 raft_consensus.cc:1261] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 1 FOLLOWER]: Refusing update from remote peer 8b1f4277056c48bab01fc9555065917a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:23.247951 5777 consensus_queue.cc:1035] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc8eccc7ae046b381dfb481007d49ea" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 44741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:23.248322 5775 consensus_queue.cc:1035] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc1914e68bd4bac8069131e1a5234d9" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:23.277076 5296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:23.277393 5296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:23.280923 5296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc76d86070df4cbb88b0001548a070a3: Sending DeleteTablet for 3 replicas of tablet 25ece7c276d04ccfba9572aa00a3d424
I20260926 19:27:23.281980 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:40602
I20260926 19:27:23.282064 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:55268
I20260926 19:27:23.282449 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ece7c276d04ccfba9572aa00a3d424 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:57736
I20260926 19:27:23.284951 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5357
I20260926 19:27:23.294360 5790 tablet_replica.cc:330] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: stopping tablet replica
I20260926 19:27:23.295459 5790 raft_consensus.cc:2229] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:23.296758 5790 raft_consensus.cc:2258] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:23.297883 5792 tablet_replica.cc:330] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: stopping tablet replica
I20260926 19:27:23.298401 5792 raft_consensus.cc:2229] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:23.298954 5792 raft_consensus.cc:2258] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:23.299695 5791 tablet_replica.cc:330] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: stopping tablet replica
I20260926 19:27:23.299916 5790 ts_tablet_manager.cc:1905] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:23.300159 5791 raft_consensus.cc:2229] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:23.300678 5791 raft_consensus.cc:2258] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:23.300932 5792 ts_tablet_manager.cc:1905] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:23.302847 5791 ts_tablet_manager.cc:1905] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:23.306883 5790 ts_tablet_manager.cc:1918] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:23.307103 5790 log.cc:1196] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/25ece7c276d04ccfba9572aa00a3d424
I20260926 19:27:23.307689 5790 ts_tablet_manager.cc:1939] T 25ece7c276d04ccfba9572aa00a3d424 P 8b1f4277056c48bab01fc9555065917a: Deleting consensus metadata
I20260926 19:27:23.308267 5792 ts_tablet_manager.cc:1918] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:23.308495 5792 log.cc:1196] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/25ece7c276d04ccfba9572aa00a3d424
I20260926 19:27:23.309087 5792 ts_tablet_manager.cc:1939] T 25ece7c276d04ccfba9572aa00a3d424 P bbc1914e68bd4bac8069131e1a5234d9: Deleting consensus metadata
Sep 26 19:27:23 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:23.309638 5283 catalog_manager.cc:4841] TS 8b1f4277056c48bab01fc9555065917a (127.1.39.3:38113): tablet 25ece7c276d04ccfba9572aa00a3d424 (table test-table [id=c00166050f8445edbf65e52dc07a3c56]) successfully deleted
I20260926 19:27:23.309890 5791 ts_tablet_manager.cc:1918] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:23.310170 5791 log.cc:1196] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/25ece7c276d04ccfba9572aa00a3d424
I20260926 19:27:23.310717 5791 ts_tablet_manager.cc:1939] T 25ece7c276d04ccfba9572aa00a3d424 P fbc8eccc7ae046b381dfb481007d49ea: Deleting consensus metadata
I20260926 19:27:23.311661 5281 catalog_manager.cc:4841] TS bbc1914e68bd4bac8069131e1a5234d9 (127.1.39.2:42307): tablet 25ece7c276d04ccfba9572aa00a3d424 (table test-table [id=c00166050f8445edbf65e52dc07a3c56]) successfully deleted
I20260926 19:27:23.312423 5282 catalog_manager.cc:4841] TS fbc8eccc7ae046b381dfb481007d49ea (127.1.39.1:44741): tablet 25ece7c276d04ccfba9572aa00a3d424 (table test-table [id=c00166050f8445edbf65e52dc07a3c56]) successfully deleted
I20260926 19:27:23.316778 5764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.286688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.1.39.1:44741)
0926 19:27:23.286905 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.286914 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.289835 (+ 2921us) server_negotiation.cc:407] Connection header received
0926 19:27:23.290166 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.290176 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.290277 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.290366 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:23.301125 (+ 10759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.302339 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.304382 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.305040 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.305152 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.310695 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:23.310715 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:23.310732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:23.310814 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:23.313928 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.314735 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.314742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.314744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.314864 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.315258 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.315261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.315263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.315588 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:23.315811 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.316205 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.316417 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:27:23.317492 5479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:23.425995 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5357
I20260926 19:27:23.449159 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5490
Sep 26 19:27:23 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:23.464732 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.450000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.1.39.2:42307)
0926 19:27:23.450212 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.450218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.450237 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:23.450350 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.450355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.450437 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.450545 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:23.451807 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.452574 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.453895 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.454303 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.454440 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.458656 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:23.458671 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:23.458687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:23.458725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:23.461802 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.462619 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.462631 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.462635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.462767 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.463181 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.463185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.463187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.463551 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:23.463857 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.464142 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.464366 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 19:27:23.465526 5612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:23.561422 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5490
I20260926 19:27:23.579298 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5622
Sep 26 19:27:23 dist-test-slave-9wb0 krb5kdc[5252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:23.598482 5763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:23.580059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40620 (local address 127.1.39.3:38113)
0926 19:27:23.580264 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:23.580271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:23.580319 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:27:23.580499 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:23.580507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:23.580610 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:23.580729 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:23.582091 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.583350 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.584817 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:23.585638 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:23.585770 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:23.591058 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:23.591073 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:23.591094 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:23.591137 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:23.595221 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.596019 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.596031 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.596036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.596211 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:23.596633 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:23.596638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:23.596641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:23.597106 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:23.597478 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:23.597828 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 19:27:23.598062 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 19:27:23.599413 5744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:23.722450 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5622
I20260926 19:27:23.742125 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5265
I20260926 19:27:23.743218 5326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:23.849431 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5265
2026-09-26T19:27:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4242 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T19:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:23Z Disabled control of system clock
I20260926 19:27:23.916527 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:32997
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:32997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 19:27:24.232430 5801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:24.232721 5801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:24.232805 5801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:24.239974 5801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:24.240095 5801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:24.240134 5801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:24.240182 5801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:24.240221 5801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:24.250118 5801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:32997
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:32997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:24.251374 5801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:24.253228 5801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:24.264822 5806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:24.264739 5809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:24.264789 5807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:24.265774 5801 server_base.cc:1034] running on GCE node
I20260926 19:27:24.266654 5801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:24.268209 5801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:24.269402 5801 hybrid_clock.cc:648] HybridClock initialized: now 1790450844269354 us; error 48 us; skew 500 ppm
I20260926 19:27:24.269783 5801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:24.272179 5801 webserver.cc:460] Webserver started at http://127.1.39.62:33997/ using document root <none> and password file <none>
I20260926 19:27:24.272873 5801 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:24.272960 5801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:24.273391 5801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:24.276239 5801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "99543410d85f492983ff583bc71d7f03"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "c9bac8421f7ae4b48057047da75335c3"
server_key_iv: "2309be101e8e2fe392ac86dd8f70f116"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.277015 5801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "99543410d85f492983ff583bc71d7f03"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "c9bac8421f7ae4b48057047da75335c3"
server_key_iv: "2309be101e8e2fe392ac86dd8f70f116"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.282543 5801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:27:24.286316 5815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:24.296864 5801 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.011s sys 0.001s
I20260926 19:27:24.297123 5801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "99543410d85f492983ff583bc71d7f03"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "c9bac8421f7ae4b48057047da75335c3"
server_key_iv: "2309be101e8e2fe392ac86dd8f70f116"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.297472 5801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:24.304155 5801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:24.305073 5801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:24.305423 5801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:24.324417 5801 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:32997
I20260926 19:27:24.324450 5866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:32997 every 8 connection(s)
I20260926 19:27:24.325836 5801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:24.326327 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5801
I20260926 19:27:24.326543 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:24.327384 1180 external_mini_cluster.cc:1428] Setting key e390e2683550ce9eaa7d2e578d791fe9
I20260926 19:27:24.330338 5867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:24.335906 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:24.329519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.1.39.62:32997)
0926 19:27:24.330567 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:24.330579 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:24.330623 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:27:24.332120 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:24.332140 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:24.332538 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:24.332926 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:24.334066 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:24.334072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:24.334084 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:24.334098 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:24.334169 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:24.334184 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:24.334433 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:24.334608 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 19:27:24.334849 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":354,"threads_started":1}
I20260926 19:27:24.341673 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: Bootstrap starting.
I20260926 19:27:24.345165 5867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:24.346385 5867 log.cc:824] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:24.349479 5867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: No bootstrap required, opened a new log
I20260926 19:27:24.356667 5867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } }
I20260926 19:27:24.357023 5867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:24.357118 5867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99543410d85f492983ff583bc71d7f03, State: Initialized, Role: FOLLOWER
I20260926 19:27:24.357700 5867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } }
I20260926 19:27:24.357921 5867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:24.358060 5867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:24.358234 5867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:24.360462 5867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } }
I20260926 19:27:24.360942 5867 leader_election.cc:302] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99543410d85f492983ff583bc71d7f03; no voters:
I20260926 19:27:24.361568 5867 leader_election.cc:288] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:24.361670 5872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:24.362385 5872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [term 1 LEADER]: Becoming Leader. State: Replica: 99543410d85f492983ff583bc71d7f03, State: Running, Role: LEADER
I20260926 19:27:24.362879 5872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } }
I20260926 19:27:24.363615 5867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:24.365625 5874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99543410d85f492983ff583bc71d7f03. Latest consensus state: current_term: 1 leader_uuid: "99543410d85f492983ff583bc71d7f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } } }
I20260926 19:27:24.365597 5873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99543410d85f492983ff583bc71d7f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99543410d85f492983ff583bc71d7f03" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32997 } } }
I20260926 19:27:24.365998 5874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:24.365998 5873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:24.368069 5878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:24.372550 5878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:24.381542 5878 catalog_manager.cc:1347] Generated new cluster ID: 4f7f2f4957f444f891957852e1d3f94b
I20260926 19:27:24.381657 5878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:24.390779 5878 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:24.391229 5878 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:24.402926 5878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: Generated new TSK 0
I20260926 19:27:24.403851 5878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:27:24.411409 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--builtin_ntp_servers=127.1.39.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 19:27:24.739652 5891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:24.739917 5891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:24.740062 5891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:24.747335 5891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:24.747431 5891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:24.747596 5891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:24.756728 5891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:24.758029 5891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:24.759850 5891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:24.773517 5897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:24.773625 5896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:24.773581 5899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:24.774516 5891 server_base.cc:1034] running on GCE node
I20260926 19:27:24.775126 5891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:24.776101 5891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:24.777364 5891 hybrid_clock.cc:648] HybridClock initialized: now 1790450844777315 us; error 36 us; skew 500 ppm
I20260926 19:27:24.777755 5891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:24.784365 5891 webserver.cc:460] Webserver started at http://127.1.39.1:35997/ using document root <none> and password file <none>
I20260926 19:27:24.785096 5891 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:24.785233 5891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:24.785590 5891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:24.788444 5891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "812c8b42d7ff46cd93429c3253b00b86"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "279acd684bf6680217ceda792382aad2"
server_key_iv: "0c4a22a472869b4caef3b91ac347016f"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.789413 5891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "812c8b42d7ff46cd93429c3253b00b86"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "279acd684bf6680217ceda792382aad2"
server_key_iv: "0c4a22a472869b4caef3b91ac347016f"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.794636 5891 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:27:24.798017 5905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:24.799256 5891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:24.799434 5891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "812c8b42d7ff46cd93429c3253b00b86"
format_stamp: "Formatted at 2026-09-26 19:27:24 on dist-test-slave-9wb0"
server_key: "279acd684bf6680217ceda792382aad2"
server_key_iv: "0c4a22a472869b4caef3b91ac347016f"
server_key_version: "encryptionkey@0"
I20260926 19:27:24.799678 5891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:24.805388 5891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:24.806262 5891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:24.806535 5891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:24.807410 5891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:24.808754 5891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:24.808879 5891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:24.809018 5891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:24.809108 5891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:24.841542 5891 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:45143
I20260926 19:27:24.841570 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:45143 every 8 connection(s)
I20260926 19:27:24.842869 5891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:24.850826 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:24.845416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57461 (local address 127.1.39.62:32997)
0926 19:27:24.845857 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:24.845862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:24.846918 (+ 1056us) server_negotiation.cc:407] Connection header received
0926 19:27:24.848583 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:24.848588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:24.848665 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:24.848779 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:24.850050 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:24.850056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:24.850061 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:24.850078 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:24.850126 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:24.850150 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:24.850268 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:24.850476 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 19:27:24.850539 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":240,"threads_started":1}
I20260926 19:27:24.851158 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 5891
I20260926 19:27:24.851343 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:24.851770 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:24.845912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:32997 (local address 127.1.39.1:57461)
0926 19:27:24.846763 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:24.846803 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:24.848329 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:24.849013 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:24.849021 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:24.849107 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:24.849761 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:24.849764 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:24.849800 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:24.849879 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:24.850248 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:24.850257 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:24.850265 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:27:24.850534 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 19:27:24.850769 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":247,"threads_started":1}
I20260926 19:27:24.852046 1180 external_mini_cluster.cc:1428] Setting key 0db0e74261dc42283de4f05309a880f8
I20260926 19:27:24.853372 6018 heartbeater.cc:347] Connected to a master server at 127.1.39.62:32997
I20260926 19:27:24.853668 6018 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:24.854305 6018 heartbeater.cc:511] Master 127.1.39.62:32997 requested a full tablet report, sending...
I20260926 19:27:24.854943 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--builtin_ntp_servers=127.1.39.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 19:27:24.856666 5832 ts_manager.cc:194] Registered new tserver with Master: 812c8b42d7ff46cd93429c3253b00b86 (127.1.39.1:45143)
W20260926 19:27:25.173929 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:25.174252 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:25.174407 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:25.181838 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:25.181982 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:25.182161 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:25.191533 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:25.192888 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:25.194780 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:25.208530 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:25.208647 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:25.208581 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:25.209934 6022 server_base.cc:1034] running on GCE node
I20260926 19:27:25.210743 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:25.211829 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:25.213096 6022 hybrid_clock.cc:648] HybridClock initialized: now 1790450845213055 us; error 69 us; skew 500 ppm
I20260926 19:27:25.213557 6022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:25.216280 6022 webserver.cc:460] Webserver started at http://127.1.39.2:46537/ using document root <none> and password file <none>
I20260926 19:27:25.217062 6022 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:25.217207 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:25.217603 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:25.221454 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "6cb332c0ec95019e75c6fb8db1239193"
server_key_iv: "24bf700a0a169cbad1ff3fb9fe7b236c"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.222638 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "6cb332c0ec95019e75c6fb8db1239193"
server_key_iv: "24bf700a0a169cbad1ff3fb9fe7b236c"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.228708 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 19:27:25.232719 6036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.234258 6022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:25.234454 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "6cb332c0ec95019e75c6fb8db1239193"
server_key_iv: "24bf700a0a169cbad1ff3fb9fe7b236c"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.234699 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:25.240502 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:25.241359 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:25.241654 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:25.242558 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:25.243942 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:25.244058 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.244192 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:25.244277 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.276185 6022 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:34791
I20260926 19:27:25.276228 6148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:34791 every 8 connection(s)
I20260926 19:27:25.277649 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:25.285388 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6022
I20260926 19:27:25.285564 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:25.286191 1180 external_mini_cluster.cc:1428] Setting key 469918eac6bf2bb45fecd1a79b09bbb9
I20260926 19:27:25.286469 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.280302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:51135 (local address 127.1.39.62:32997)
0926 19:27:25.280507 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.280514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.282028 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 19:27:25.284174 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.284179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.284228 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.284314 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.285670 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.285677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.285682 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.285702 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.285741 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.285756 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.285885 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.286102 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.286165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:27:25.287323 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.280904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:32997 (local address 127.1.39.2:51135)
0926 19:27:25.281853 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.281898 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.283933 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.284519 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.284527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.284614 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.285323 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.285329 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.285379 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.285479 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.285852 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.285860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.285865 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.286146 (+ 281us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.286330 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":255,"threads_started":1}
I20260926 19:27:25.288947 6149 heartbeater.cc:347] Connected to a master server at 127.1.39.62:32997
I20260926 19:27:25.289250 6149 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:25.289248 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--builtin_ntp_servers=127.1.39.20:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 19:27:25.289886 6149 heartbeater.cc:511] Master 127.1.39.62:32997 requested a full tablet report, sending...
I20260926 19:27:25.291451 5832 ts_manager.cc:194] Registered new tserver with Master: 798fd38ee7ea4f9ba43ea6691a5c46a0 (127.1.39.2:34791)
W20260926 19:27:25.630887 6152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:25.631178 6152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:25.631335 6152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:25.638666 6152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:25.638785 6152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:25.638949 6152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:25.648196 6152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:32997
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:25.649607 6152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:25.651492 6152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:25.664669 6157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:25.664628 6158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:25.664634 6160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:25.666328 6152 server_base.cc:1034] running on GCE node
I20260926 19:27:25.666914 6152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:25.667783 6152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:25.669003 6152 hybrid_clock.cc:648] HybridClock initialized: now 1790450845668952 us; error 62 us; skew 500 ppm
I20260926 19:27:25.669361 6152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:25.671602 6152 webserver.cc:460] Webserver started at http://127.1.39.3:43051/ using document root <none> and password file <none>
I20260926 19:27:25.672317 6152 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:25.672443 6152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:25.672744 6152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:25.675612 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "90020f753cb24446ac2f6612c50f64dc"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "f55dde4d680184e6deeaa0300f2ddfd4"
server_key_iv: "8191669c17b9b42e2bd9075dafcb3634"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.676378 6152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "90020f753cb24446ac2f6612c50f64dc"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "f55dde4d680184e6deeaa0300f2ddfd4"
server_key_iv: "8191669c17b9b42e2bd9075dafcb3634"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.681280 6152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 19:27:25.684492 6166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.685761 6152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:25.685925 6152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "90020f753cb24446ac2f6612c50f64dc"
format_stamp: "Formatted at 2026-09-26 19:27:25 on dist-test-slave-9wb0"
server_key: "f55dde4d680184e6deeaa0300f2ddfd4"
server_key_iv: "8191669c17b9b42e2bd9075dafcb3634"
server_key_version: "encryptionkey@0"
I20260926 19:27:25.686149 6152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:25.691711 6152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:25.692457 6152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:25.692719 6152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:25.693677 6152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:25.695039 6152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:25.695164 6152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.695292 6152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:25.695379 6152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:25.729077 6152 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:33809
I20260926 19:27:25.729137 6278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:33809 every 8 connection(s)
I20260926 19:27:25.730489 6152 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:25.731037 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6152
I20260926 19:27:25.731233 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:25.731904 1180 external_mini_cluster.cc:1428] Setting key df77f467422baeccf4c08a1a2507f5fe
I20260926 19:27:25.739172 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.733274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:39083 (local address 127.1.39.62:32997)
0926 19:27:25.733497 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.733504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.735261 (+ 1757us) server_negotiation.cc:407] Connection header received
0926 19:27:25.736841 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.736845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.736893 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.736973 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.738490 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.738495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.738498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.738514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.738545 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.738556 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.738678 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.738895 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.738942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 19:27:25.740206 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.734052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:32997 (local address 127.1.39.3:39083)
0926 19:27:25.735106 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.735151 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.736613 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.737240 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.737248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.737409 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.738158 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.738161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.738198 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.738283 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.738662 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.738674 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.738685 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.738967 (+ 282us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.739276 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":332,"threads_started":1}
I20260926 19:27:25.741537 6279 heartbeater.cc:347] Connected to a master server at 127.1.39.62:32997
I20260926 19:27:25.741804 6279 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:25.742313 6279 heartbeater.cc:511] Master 127.1.39.62:32997 requested a full tablet report, sending...
I20260926 19:27:25.743674 5832 ts_manager.cc:194] Registered new tserver with Master: 90020f753cb24446ac2f6612c50f64dc (127.1.39.3:33809)
I20260926 19:27:25.747296 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:27:25.755143 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.752250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.1.39.62:32997)
0926 19:27:25.752449 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.752459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.753059 (+ 600us) server_negotiation.cc:407] Connection header received
0926 19:27:25.753228 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.753233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.753287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.753506 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.754525 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.754530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.754533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.754548 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.754606 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.754618 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.754728 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.754890 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.754937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 19:27:25.759086 5832 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:25.765908 5832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:25.795289 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.778307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53474 (local address 127.1.39.2:34791)
0926 19:27:25.787649 (+ 9342us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.787665 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.792175 (+ 4510us) server_negotiation.cc:407] Connection header received
0926 19:27:25.792260 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.792270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.792561 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.793379 (+ 818us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.794359 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.794389 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.794407 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.794435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.794507 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.794530 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.794672 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.794826 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.794934 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9000,"thread_start_us":8736,"threads_started":1}
I20260926 19:27:25.795424 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.777971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:34791 (local address 127.0.0.1:53474)
0926 19:27:25.791584 (+ 13613us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.791606 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.791731 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.792772 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.792783 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.792876 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.794060 (+ 1184us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.794065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.794102 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.794180 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.794646 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.794656 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.794664 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.794835 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.794924 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13409,"thread_start_us":11413,"threads_started":1}
I20260926 19:27:25.796006 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.781640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.1.39.3:33809)
0926 19:27:25.791015 (+ 9375us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.791024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.792157 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 19:27:25.792401 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.792408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.792558 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.792700 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.794778 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.794786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.794798 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.794815 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.794865 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.794883 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.795038 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.795541 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.795628 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9152,"thread_start_us":8978,"threads_started":1}
I20260926 19:27:25.796130 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.777778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:45143 (local address 127.0.0.1:45466)
0926 19:27:25.790374 (+ 12596us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.790429 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.790612 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.793569 (+ 2957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.793575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.793599 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.794404 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.794409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.794426 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.794470 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.795502 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.795506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.795511 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.795652 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.795699 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11758,"thread_start_us":13821,"threads_started":1}
I20260926 19:27:25.796409 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.777945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.1.39.1:45143)
0926 19:27:25.792185 (+ 14240us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.792193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.792222 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:27:25.792285 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.792292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.792558 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.792705 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.794632 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.794640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.794652 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.794671 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.794729 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.795140 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.795273 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.796015 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.796117 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14092,"thread_start_us":13976,"threads_started":1}
I20260926 19:27:25.796739 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.780857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:33809 (local address 127.0.0.1:59384)
0926 19:27:25.791937 (+ 11080us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.792058 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.792247 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.793863 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.793867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.793888 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.794546 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.794551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.794568 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.794613 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.795093 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.795098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.795103 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.795239 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.795303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10923,"thread_start_us":10337,"threads_started":1}
I20260926 19:27:25.799657 6214 tablet_service.cc:1467] Processing CreateTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa (DEFAULT_TABLE table=test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:25.799700 6084 tablet_service.cc:1467] Processing CreateTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa (DEFAULT_TABLE table=test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:25.800100 5953 tablet_service.cc:1467] Processing CreateTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa (DEFAULT_TABLE table=test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:25.801033 6214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5056d2b36de5449c84e1817a4e0b8baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:25.801115 6084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5056d2b36de5449c84e1817a4e0b8baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:25.801219 5953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5056d2b36de5449c84e1817a4e0b8baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:25.813547 6297 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Bootstrap starting.
I20260926 19:27:25.815641 6298 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Bootstrap starting.
I20260926 19:27:25.817430 6299 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Bootstrap starting.
I20260926 19:27:25.819206 6297 tablet_bootstrap.cc:654] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:25.819542 6298 tablet_bootstrap.cc:654] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:25.820168 6299 tablet_bootstrap.cc:654] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:25.820536 6297 log.cc:824] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:25.820642 6298 log.cc:824] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:25.821338 6299 log.cc:824] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:25.823927 6298 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: No bootstrap required, opened a new log
I20260926 19:27:25.823926 6297 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: No bootstrap required, opened a new log
I20260926 19:27:25.824227 6298 ts_tablet_manager.cc:1397] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:27:25.824229 6297 ts_tablet_manager.cc:1397] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260926 19:27:25.831656 6299 tablet_bootstrap.cc:492] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: No bootstrap required, opened a new log
I20260926 19:27:25.832000 6299 ts_tablet_manager.cc:1397] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20260926 19:27:25.834995 6297 raft_consensus.cc:348] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.834987 6298 raft_consensus.cc:348] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.835454 6298 raft_consensus.cc:374] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:25.835454 6297 raft_consensus.cc:374] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:25.835566 6297 raft_consensus.cc:729] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90020f753cb24446ac2f6612c50f64dc, State: Initialized, Role: FOLLOWER
I20260926 19:27:25.835611 6298 raft_consensus.cc:729] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798fd38ee7ea4f9ba43ea6691a5c46a0, State: Initialized, Role: FOLLOWER
I20260926 19:27:25.836292 6297 consensus_queue.cc:260] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.836287 6298 consensus_queue.cc:260] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.837929 6149 heartbeater.cc:503] Master 127.1.39.62:32997 was elected leader, sending a full tablet report...
I20260926 19:27:25.837962 6279 heartbeater.cc:503] Master 127.1.39.62:32997 was elected leader, sending a full tablet report...
I20260926 19:27:25.838510 6298 ts_tablet_manager.cc:1428] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260926 19:27:25.839406 6297 ts_tablet_manager.cc:1428] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260926 19:27:25.841030 6299 raft_consensus.cc:348] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.841547 6299 raft_consensus.cc:374] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:25.841670 6299 raft_consensus.cc:729] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 812c8b42d7ff46cd93429c3253b00b86, State: Initialized, Role: FOLLOWER
I20260926 19:27:25.842480 6299 consensus_queue.cc:260] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.844877 6018 heartbeater.cc:503] Master 127.1.39.62:32997 was elected leader, sending a full tablet report...
I20260926 19:27:25.845382 6299 ts_tablet_manager.cc:1428] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260926 19:27:25.902798 6304 raft_consensus.cc:482] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:25.903122 6304 raft_consensus.cc:504] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.904945 6304 leader_election.cc:288] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 812c8b42d7ff46cd93429c3253b00b86 (127.1.39.1:45143), 798fd38ee7ea4f9ba43ea6691a5c46a0 (127.1.39.2:34791)
I20260926 19:27:25.907675 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.905418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:37249 (local address 127.1.39.1:45143)
0926 19:27:25.905631 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.905640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.905771 (+ 131us) server_negotiation.cc:407] Connection header received
0926 19:27:25.905938 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.905946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.906005 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.906111 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.907070 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.907074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.907078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.907092 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.907119 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.907130 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.907219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.907364 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.907425 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:27:25.907686 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.905361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:45143 (local address 127.1.39.3:37249)
0926 19:27:25.905657 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.905679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.905796 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.906145 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.906152 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.906179 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.906875 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.906878 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.906895 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.906936 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.907228 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.907234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.907241 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.907339 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.907379 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 19:27:25.908594 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.905564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:44905 (local address 127.1.39.2:34791)
0926 19:27:25.905732 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.905739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.906437 (+ 698us) server_negotiation.cc:407] Connection header received
0926 19:27:25.906678 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.906683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.906743 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.906851 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.907852 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.907858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.907865 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.907884 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.907942 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.907958 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.908041 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.908255 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.908299 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:27:25.908619 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.905467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:34791 (local address 127.1.39.3:44905)
0926 19:27:25.906313 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:25.906331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:25.906490 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:25.906886 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:25.906889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:25.906910 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:25.907619 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:27:25.907622 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:25.907637 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:27:25.907682 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:25.908128 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:25.908133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:25.908138 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:27:25.908249 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 19:27:25.908292 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":515,"threads_started":1}
I20260926 19:27:25.908914 5973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5056d2b36de5449c84e1817a4e0b8baa" candidate_uuid: "90020f753cb24446ac2f6612c50f64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812c8b42d7ff46cd93429c3253b00b86" is_pre_election: true
I20260926 19:27:25.909524 5973 raft_consensus.cc:2454] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90020f753cb24446ac2f6612c50f64dc in term 0.
I20260926 19:27:25.909584 6104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5056d2b36de5449c84e1817a4e0b8baa" candidate_uuid: "90020f753cb24446ac2f6612c50f64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" is_pre_election: true
I20260926 19:27:25.910053 6104 raft_consensus.cc:2454] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90020f753cb24446ac2f6612c50f64dc in term 0.
I20260926 19:27:25.910250 6170 leader_election.cc:302] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 812c8b42d7ff46cd93429c3253b00b86, 90020f753cb24446ac2f6612c50f64dc; no voters:
I20260926 19:27:25.910662 6304 raft_consensus.cc:2781] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:25.910818 6304 raft_consensus.cc:482] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:25.910920 6304 raft_consensus.cc:3037] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:25.912997 6304 raft_consensus.cc:504] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.913611 6304 leader_election.cc:288] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [CANDIDATE]: Term 1 election: Requested vote from peers 812c8b42d7ff46cd93429c3253b00b86 (127.1.39.1:45143), 798fd38ee7ea4f9ba43ea6691a5c46a0 (127.1.39.2:34791)
I20260926 19:27:25.913945 5973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5056d2b36de5449c84e1817a4e0b8baa" candidate_uuid: "90020f753cb24446ac2f6612c50f64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "812c8b42d7ff46cd93429c3253b00b86"
I20260926 19:27:25.914072 6104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5056d2b36de5449c84e1817a4e0b8baa" candidate_uuid: "90020f753cb24446ac2f6612c50f64dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0"
I20260926 19:27:25.914145 5973 raft_consensus.cc:3037] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:25.914289 6104 raft_consensus.cc:3037] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:25.916661 5973 raft_consensus.cc:2454] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90020f753cb24446ac2f6612c50f64dc in term 1.
I20260926 19:27:25.916913 6104 raft_consensus.cc:2454] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90020f753cb24446ac2f6612c50f64dc in term 1.
I20260926 19:27:25.917018 6170 leader_election.cc:302] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 812c8b42d7ff46cd93429c3253b00b86, 90020f753cb24446ac2f6612c50f64dc; no voters:
I20260926 19:27:25.917318 6304 raft_consensus.cc:2781] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:25.917753 6304 raft_consensus.cc:686] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 1 LEADER]: Becoming Leader. State: Replica: 90020f753cb24446ac2f6612c50f64dc, State: Running, Role: LEADER
I20260926 19:27:25.918186 6304 consensus_queue.cc:237] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } }
I20260926 19:27:25.922324 5831 catalog_manager.cc:5495] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 90020f753cb24446ac2f6612c50f64dc (127.1.39.3). New cstate: current_term: 1 leader_uuid: "90020f753cb24446ac2f6612c50f64dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90020f753cb24446ac2f6612c50f64dc" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33809 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:25.955451 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:25.953400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.1.39.3:33809)
0926 19:27:25.953635 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:25.953642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:25.953659 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:27:25.953753 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:25.953757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:25.953830 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:25.953940 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:25.954856 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:25.954863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:25.954871 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:25.954887 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:25.954915 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:25.954926 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:25.955052 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:25.955143 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 19:27:25.955201 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 19:27:25.964047 6104 raft_consensus.cc:1261] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 1 FOLLOWER]: Refusing update from remote peer 90020f753cb24446ac2f6612c50f64dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:25.964051 5973 raft_consensus.cc:1261] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 1 FOLLOWER]: Refusing update from remote peer 90020f753cb24446ac2f6612c50f64dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:25.965102 6307 consensus_queue.cc:1035] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "812c8b42d7ff46cd93429c3253b00b86" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:25.965478 6304 consensus_queue.cc:1035] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "798fd38ee7ea4f9ba43ea6691a5c46a0" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 34791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:25.991549 5832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:25.991788 5832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:25.995126 5832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99543410d85f492983ff583bc71d7f03: Sending DeleteTablet for 3 replicas of tablet 5056d2b36de5449c84e1817a4e0b8baa
I20260926 19:27:25.996171 5953 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:25 UTC) from {username='slave'} at 127.0.0.1:45466
I20260926 19:27:25.996238 6214 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:25 UTC) from {username='slave'} at 127.0.0.1:59384
I20260926 19:27:25.996353 6084 tablet_service.cc:1514] Processing DeleteTablet for tablet 5056d2b36de5449c84e1817a4e0b8baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:25 UTC) from {username='slave'} at 127.0.0.1:53474
I20260926 19:27:25.997274 6319 tablet_replica.cc:330] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: stopping tablet replica
I20260926 19:27:25.997788 6321 tablet_replica.cc:330] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: stopping tablet replica
I20260926 19:27:25.997782 6319 raft_consensus.cc:2229] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:25.997994 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5891
I20260926 19:27:25.998286 6320 tablet_replica.cc:330] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: stopping tablet replica
I20260926 19:27:25.998451 6319 raft_consensus.cc:2258] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:25.998669 6320 raft_consensus.cc:2229] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:25.999044 6321 raft_consensus.cc:2229] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:25.999138 6320 raft_consensus.cc:2258] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:25.999496 6321 raft_consensus.cc:2258] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:26.002051 6320 ts_tablet_manager.cc:1905] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:26.002223 6321 ts_tablet_manager.cc:1905] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:26.004045 6319 ts_tablet_manager.cc:1905] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:26.009021 6320 ts_tablet_manager.cc:1918] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:26.009235 6320 log.cc:1196] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/5056d2b36de5449c84e1817a4e0b8baa
I20260926 19:27:26.009529 6321 ts_tablet_manager.cc:1918] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:26.009759 6321 log.cc:1196] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/5056d2b36de5449c84e1817a4e0b8baa
I20260926 19:27:26.009863 6320 ts_tablet_manager.cc:1939] T 5056d2b36de5449c84e1817a4e0b8baa P 812c8b42d7ff46cd93429c3253b00b86: Deleting consensus metadata
I20260926 19:27:26.010329 6321 ts_tablet_manager.cc:1939] T 5056d2b36de5449c84e1817a4e0b8baa P 798fd38ee7ea4f9ba43ea6691a5c46a0: Deleting consensus metadata
I20260926 19:27:26.011788 5819 catalog_manager.cc:4841] TS 812c8b42d7ff46cd93429c3253b00b86 (127.1.39.1:45143): tablet 5056d2b36de5449c84e1817a4e0b8baa (table test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]) successfully deleted
I20260926 19:27:26.012012 5818 catalog_manager.cc:4841] TS 798fd38ee7ea4f9ba43ea6691a5c46a0 (127.1.39.2:34791): tablet 5056d2b36de5449c84e1817a4e0b8baa (table test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]) successfully deleted
I20260926 19:27:26.014524 6319 ts_tablet_manager.cc:1918] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:26.014739 6319 log.cc:1196] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/5056d2b36de5449c84e1817a4e0b8baa
I20260926 19:27:26.015357 6319 ts_tablet_manager.cc:1939] T 5056d2b36de5449c84e1817a4e0b8baa P 90020f753cb24446ac2f6612c50f64dc: Deleting consensus metadata
I20260926 19:27:26.016279 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:26.000526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.1.39.1:45143)
0926 19:27:26.000687 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:26.000693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:26.013872 (+ 13179us) server_negotiation.cc:407] Connection header received
0926 19:27:26.014052 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:26.014059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:26.014177 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:26.014314 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:26.015308 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:26.015318 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:26.015324 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:26.015351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:26.015394 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:26.015414 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:26.015573 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:26.015844 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 19:27:26.015927 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:27:26.017364 6013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:26.017465 5816 catalog_manager.cc:4841] TS 90020f753cb24446ac2f6612c50f64dc (127.1.39.3:33809): tablet 5056d2b36de5449c84e1817a4e0b8baa (table test-table [id=6fe7a24fa418438ea6a2f3d1f3d6284d]) successfully deleted
I20260926 19:27:26.106541 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5891
I20260926 19:27:26.130067 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6022
I20260926 19:27:26.132993 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:26.130903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.1.39.2:34791)
0926 19:27:26.131067 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:26.131071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:26.131089 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:27:26.131289 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:26.131292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:26.131380 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:26.131465 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:26.132353 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:26.132357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:26.132360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:26.132374 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:26.132402 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:26.132414 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:26.132507 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:26.132635 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 19:27:26.132677 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:27:26.133876 6144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:26.224843 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6022
I20260926 19:27:26.252652 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6152
I20260926 19:27:26.256175 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:26.253551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.1.39.3:33809)
0926 19:27:26.253748 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:26.253753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:26.253909 (+ 156us) server_negotiation.cc:407] Connection header received
0926 19:27:26.254117 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:26.254122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:26.254180 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:26.254277 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:26.255346 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:26.255351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:26.255354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:26.255381 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:26.255420 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:26.255432 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:26.255644 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:26.255788 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 19:27:26.255856 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:27:26.256987 6274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:26.379058 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6152
I20260926 19:27:26.398051 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 5801
I20260926 19:27:26.399133 5862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:26.505276 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 5801
2026-09-26T19:27:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2654 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 19:27:26.546969 1180 mini_oidc.cc:152] Starting JWKS server
I20260926 19:27:26.549715 1180 webserver.cc:460] Webserver started at https://127.0.0.1:43653/ using document root <none> and password file <none>
I20260926 19:27:26.550096 1180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:27:26.550989 1180 webserver.cc:460] Webserver started at http://127.0.0.1:43561/ using document root <none> and password file <none>
2026-09-26T19:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:26Z Disabled control of system clock
I20260926 19:27:26.579499 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46847
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:45913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:46847
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/testchainca.cert with env {}
W20260926 19:27:26.897991 6332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:26.898332 6332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:26.898497 6332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:26.905547 6332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:26.905689 6332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:26.905778 6332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:26.905877 6332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:26.905961 6332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:27:26.906050 6332 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:27:26.906111 6332 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43653/jwks.json
I20260926 19:27:26.916052 6332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:46847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:26.917438 6332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:26.918737 6332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:26.930373 6340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:26.930353 6338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:26.930671 6337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:26.931349 6332 server_base.cc:1034] running on GCE node
I20260926 19:27:26.932195 6332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:26.933790 6332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:26.935042 6332 hybrid_clock.cc:648] HybridClock initialized: now 1790450846934995 us; error 34 us; skew 500 ppm
I20260926 19:27:26.935420 6332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:26.937674 6332 webserver.cc:460] Webserver started at http://127.1.39.62:41909/ using document root <none> and password file <none>
I20260926 19:27:26.938329 6332 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:26.938469 6332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:26.938767 6332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:26.941565 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "6a3789002d024d498bb809ee380ac22b"
format_stamp: "Formatted at 2026-09-26 19:27:26 on dist-test-slave-9wb0"
server_key: "2b10351c3e5f0628528bec1cb24e5322"
server_key_iv: "69e6be7b4185519b1fb737d23cc2d01f"
server_key_version: "encryptionkey@0"
I20260926 19:27:26.942320 6332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "6a3789002d024d498bb809ee380ac22b"
format_stamp: "Formatted at 2026-09-26 19:27:26 on dist-test-slave-9wb0"
server_key: "2b10351c3e5f0628528bec1cb24e5322"
server_key_iv: "69e6be7b4185519b1fb737d23cc2d01f"
server_key_version: "encryptionkey@0"
I20260926 19:27:26.947222 6332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:27:26.950657 6346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:26.952184 6332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:26.952456 6332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "6a3789002d024d498bb809ee380ac22b"
format_stamp: "Formatted at 2026-09-26 19:27:26 on dist-test-slave-9wb0"
server_key: "2b10351c3e5f0628528bec1cb24e5322"
server_key_iv: "69e6be7b4185519b1fb737d23cc2d01f"
server_key_version: "encryptionkey@0"
I20260926 19:27:26.952720 6332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:26.984067 6332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:26.985035 6332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:26.985368 6332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:27.005014 6332 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:46847
I20260926 19:27:27.005036 6397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:46847 every 8 connection(s)
I20260926 19:27:27.006476 6332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:27.007364 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6332
I20260926 19:27:27.007645 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:27.008308 1180 external_mini_cluster.cc:1428] Setting key 013a1f3614752c0278a1c63698647908
I20260926 19:27:27.011381 6398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:27.025182 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:27.010438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.1.39.62:46847)
0926 19:27:27.011534 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:27.011549 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:27.011605 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:27:27.014088 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:27.014117 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:27.014781 (+ 664us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:27.015248 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:27.015836 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.018132 (+ 2296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.019599 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.020667 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.020858 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.022204 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:27.022215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:27.022235 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:27.023019 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:27.023120 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:27.023156 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:27.023290 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:27.023524 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 19:27:27.023876 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260926 19:27:27.026396 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b: Bootstrap starting.
I20260926 19:27:27.029151 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:27:27.029886 6398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:27.030931 6398 log.cc:824] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:27.034092 6398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b: No bootstrap required, opened a new log
I20260926 19:27:27.042467 6398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } }
I20260926 19:27:27.042852 6398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:27.042955 6398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a3789002d024d498bb809ee380ac22b, State: Initialized, Role: FOLLOWER
I20260926 19:27:27.043555 6398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } }
I20260926 19:27:27.043802 6398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:27.043946 6398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:27.044144 6398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:27.046525 6398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } }
I20260926 19:27:27.047077 6398 leader_election.cc:302] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a3789002d024d498bb809ee380ac22b; no voters:
I20260926 19:27:27.047734 6398 leader_election.cc:288] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:27.047907 6403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:27.049345 6403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [term 1 LEADER]: Becoming Leader. State: Replica: 6a3789002d024d498bb809ee380ac22b, State: Running, Role: LEADER
I20260926 19:27:27.049948 6403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } }
I20260926 19:27:27.050216 6398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:27.053570 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a3789002d024d498bb809ee380ac22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } } }
I20260926 19:27:27.054005 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:27.054487 6405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a3789002d024d498bb809ee380ac22b. Latest consensus state: current_term: 1 leader_uuid: "6a3789002d024d498bb809ee380ac22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a3789002d024d498bb809ee380ac22b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46847 } } }
I20260926 19:27:27.054901 6405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:27.056355 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:27.061910 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:27.072081 6412 catalog_manager.cc:1347] Generated new cluster ID: a0a590ed770d4211986149c39774473f
I20260926 19:27:27.072228 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:27.087692 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:27.089180 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:27.111040 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a3789002d024d498bb809ee380ac22b: Generated new TSK 0
I20260926 19:27:27.111974 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 19:27:27.149024 6428 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 19:27:27.149223 6428 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:27:27.150411 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:27.128765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.1.39.62:46847)
0926 19:27:27.128967 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:27.128974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:27.129589 (+ 615us) server_negotiation.cc:407] Connection header received
0926 19:27:27.129878 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:27.129884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:27.129969 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:27.130131 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:27:27.130637 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.131786 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.133146 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.133848 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.133982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.134434 (+ 452us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:27:27.149692 (+ 15258us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 19:27:27.150080 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 19:27:27.150547 6401 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56104: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 19:27:27.152129 6427 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:27.128766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46847 (local address 127.0.0.1:56104)
0926 19:27:27.129496 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:27.129512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:27.129658 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:27.130175 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:27.130179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:27.130188 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:27:27.130498 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:27.130506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.131923 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.131927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:27.133014 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:27.133024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.133964 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.134003 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:27.134132 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.134185 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:27:27.150026 (+ 15841us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 19:27:27.150451 (+ 425us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.39.62:46847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":371,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.39.62:46847: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 19:27:27.154704 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6332
I20260926 19:27:27.155933 6393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:27.252014 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6332
2026-09-26T19:27:27Z chronyd exiting
I20260926 19:27:27.302789 1180 test_util.cc:175] -----------------------------------------------
I20260926 19:27:27.302951 1180 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790450801920901-1180-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 19:27:27.306640 1180 mini_oidc.cc:152] Starting JWKS server
I20260926 19:27:27.308506 1180 webserver.cc:460] Webserver started at https://127.0.0.1:34895/ using document root <none> and password file <none>
I20260926 19:27:27.308758 1180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:27:27.309412 1180 webserver.cc:460] Webserver started at http://127.0.0.1:45341/ using document root <none> and password file <none>
2026-09-26T19:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:27Z Disabled control of system clock
I20260926 19:27:27.336491 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33943
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33943
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34895/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/testchainca.cert with env {}
W20260926 19:27:27.669219 6435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:27.669515 6435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:27.669610 6435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:27.676873 6435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:27.676986 6435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:27.677024 6435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:27.677073 6435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:27.677117 6435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:27:27.677179 6435 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:27:27.677219 6435 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34895/jwks.json
I20260926 19:27:27.687098 6435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34895/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:27.688314 6435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:27.689587 6435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:27.701642 6443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:27.701692 6441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:27.701786 6440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:27.702719 6435 server_base.cc:1034] running on GCE node
I20260926 19:27:27.703567 6435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:27.704969 6435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:27.706207 6435 hybrid_clock.cc:648] HybridClock initialized: now 1790450847706167 us; error 54 us; skew 500 ppm
I20260926 19:27:27.706600 6435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:27.708817 6435 webserver.cc:460] Webserver started at http://127.1.39.62:37169/ using document root <none> and password file <none>
I20260926 19:27:27.709628 6435 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:27.709719 6435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:27.709985 6435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:27.712882 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "b6450a9ef9c640999f3d1df62be7d1a1"
format_stamp: "Formatted at 2026-09-26 19:27:27 on dist-test-slave-9wb0"
server_key: "bde8dde97b204fb2d54cdbf206e30e0a"
server_key_iv: "75e3cb9b4e1a197366b44ae7d5838ab3"
server_key_version: "encryptionkey@0"
I20260926 19:27:27.713683 6435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "b6450a9ef9c640999f3d1df62be7d1a1"
format_stamp: "Formatted at 2026-09-26 19:27:27 on dist-test-slave-9wb0"
server_key: "bde8dde97b204fb2d54cdbf206e30e0a"
server_key_iv: "75e3cb9b4e1a197366b44ae7d5838ab3"
server_key_version: "encryptionkey@0"
I20260926 19:27:27.718454 6435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:27:27.721726 6449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:27.723026 6435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:27:27.723212 6435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "b6450a9ef9c640999f3d1df62be7d1a1"
format_stamp: "Formatted at 2026-09-26 19:27:27 on dist-test-slave-9wb0"
server_key: "bde8dde97b204fb2d54cdbf206e30e0a"
server_key_iv: "75e3cb9b4e1a197366b44ae7d5838ab3"
server_key_version: "encryptionkey@0"
I20260926 19:27:27.723435 6435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:27.752488 6435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:27.753423 6435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:27.753723 6435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:27.772737 6435 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33943
I20260926 19:27:27.772758 6500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33943 every 8 connection(s)
I20260926 19:27:27.774175 6435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:27.775055 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6435
I20260926 19:27:27.775288 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:27.776017 1180 external_mini_cluster.cc:1428] Setting key 97c2f7c3510a6598ff66f1d82cc92420
I20260926 19:27:27.778622 6501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:27.789654 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:27.777896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.1.39.62:33943)
0926 19:27:27.778958 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:27.778968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:27.779011 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:27:27.780679 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:27.780702 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:27.781178 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:27.781622 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:27.782139 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.783535 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.785388 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.786035 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.786168 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.787357 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:27.787364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:27.787376 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:27.787931 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:27.788002 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:27.788021 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:27.788117 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:27.788338 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 19:27:27.788680 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":369,"threads_started":1}
I20260926 19:27:27.790469 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1: Bootstrap starting.
I20260926 19:27:27.793241 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:27:27.793733 6501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:27.794901 6501 log.cc:824] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:27.797780 6501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1: No bootstrap required, opened a new log
I20260926 19:27:27.804987 6501 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } }
I20260926 19:27:27.805392 6501 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:27.805472 6501 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6450a9ef9c640999f3d1df62be7d1a1, State: Initialized, Role: FOLLOWER
I20260926 19:27:27.805931 6501 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } }
I20260926 19:27:27.806129 6501 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:27.806221 6501 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:27.806348 6501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:27.808322 6501 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } }
I20260926 19:27:27.808746 6501 leader_election.cc:302] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6450a9ef9c640999f3d1df62be7d1a1; no voters:
I20260926 19:27:27.809247 6501 leader_election.cc:288] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:27.809450 6506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:27.811303 6501 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:27.811756 6506 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [term 1 LEADER]: Becoming Leader. State: Replica: b6450a9ef9c640999f3d1df62be7d1a1, State: Running, Role: LEADER
I20260926 19:27:27.812307 6506 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } }
I20260926 19:27:27.815448 6507 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } } }
I20260926 19:27:27.815354 6508 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6450a9ef9c640999f3d1df62be7d1a1. Latest consensus state: current_term: 1 leader_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6450a9ef9c640999f3d1df62be7d1a1" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33943 } } }
I20260926 19:27:27.815798 6508 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:27.815798 6507 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:27.816664 6517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:27.822424 6517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:27.831854 6517 catalog_manager.cc:1347] Generated new cluster ID: 55e01367e962436c987e08605f82a704
I20260926 19:27:27.832015 6517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:27.851586 6517 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:27.852874 6517 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:27.860944 6517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6450a9ef9c640999f3d1df62be7d1a1: Generated new TSK 0
I20260926 19:27:27.862067 6517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 19:27:27.909875 6531 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 19:27:27.910012 6531 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:27:27.911109 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:27.884173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53236 (local address 127.1.39.62:33943)
0926 19:27:27.884363 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:27.884372 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:27.884795 (+ 423us) server_negotiation.cc:407] Connection header received
0926 19:27:27.885044 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:27.885048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:27.885127 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:27.885287 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:27:27.885773 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.886522 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.887881 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.888308 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.888472 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.888974 (+ 502us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:27:27.910394 (+ 21420us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 19:27:27.910773 (+ 379us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 19:27:27.911237 6504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53236: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 19:27:27.911281 6530 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:27.884070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33943 (local address 127.0.0.1:53236)
0926 19:27:27.884692 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:27.884707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:27.884870 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:27.885320 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:27.885326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:27.885339 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:27:27.885605 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:27.885613 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.886661 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.886669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:27.887739 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:27.887749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:27.888475 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:27.888550 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:27.888688 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:27.888718 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:27:27.910634 (+ 21916us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 19:27:27.910892 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.39.62:33943: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34895/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
I20260926 19:27:27.912585 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6435
I20260926 19:27:27.913830 6496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:28.006757 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6435
2026-09-26T19:27:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (755 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 19:27:28.062718 1180 mini_oidc.cc:152] Starting JWKS server
I20260926 19:27:28.064512 1180 webserver.cc:460] Webserver started at https://127.0.0.1:36235/ using document root <none> and password file <none>
I20260926 19:27:28.064708 1180 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 19:27:28.065342 1180 webserver.cc:460] Webserver started at http://127.0.0.1:36917/ using document root <none> and password file <none>
2026-09-26T19:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:28Z Disabled control of system clock
I20260926 19:27:28.092137 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33865
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36235/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 19:27:28.414487 6538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:28.414784 6538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:28.414867 6538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:28.422111 6538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:28.422219 6538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:28.422260 6538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:28.422309 6538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:28.422353 6538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:27:28.422415 6538 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:27:28.422453 6538 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36235/jwks.json
I20260926 19:27:28.431532 6538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36235/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:28.432788 6538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:28.434607 6538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:28.446673 6544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:28.446753 6546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:28.446728 6543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:28.448108 6538 server_base.cc:1034] running on GCE node
I20260926 19:27:28.448976 6538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:28.450482 6538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:28.451746 6538 hybrid_clock.cc:648] HybridClock initialized: now 1790450848451676 us; error 61 us; skew 500 ppm
I20260926 19:27:28.452168 6538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:28.454698 6538 webserver.cc:460] Webserver started at http://127.1.39.62:45871/ using document root <none> and password file <none>
I20260926 19:27:28.455469 6538 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:28.455613 6538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:28.455961 6538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:28.459116 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "3035f81b89b64da292883649c2cdfb9c"
format_stamp: "Formatted at 2026-09-26 19:27:28 on dist-test-slave-9wb0"
server_key: "3825e6800ad256039552aff1830780af"
server_key_iv: "1845b85ecf89c76e613c0785d390d720"
server_key_version: "encryptionkey@0"
I20260926 19:27:28.459997 6538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "3035f81b89b64da292883649c2cdfb9c"
format_stamp: "Formatted at 2026-09-26 19:27:28 on dist-test-slave-9wb0"
server_key: "3825e6800ad256039552aff1830780af"
server_key_iv: "1845b85ecf89c76e613c0785d390d720"
server_key_version: "encryptionkey@0"
I20260926 19:27:28.465597 6538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 19:27:28.469530 6552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:28.470947 6538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:28.471129 6538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "3035f81b89b64da292883649c2cdfb9c"
format_stamp: "Formatted at 2026-09-26 19:27:28 on dist-test-slave-9wb0"
server_key: "3825e6800ad256039552aff1830780af"
server_key_iv: "1845b85ecf89c76e613c0785d390d720"
server_key_version: "encryptionkey@0"
I20260926 19:27:28.471338 6538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:28.497253 6538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:28.498167 6538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:28.498464 6538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:28.517668 6538 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33865
I20260926 19:27:28.517691 6603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33865 every 8 connection(s)
I20260926 19:27:28.519642 6538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:28.520375 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6538
I20260926 19:27:28.520608 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:28.521450 1180 external_mini_cluster.cc:1428] Setting key 120fccaa20f87c29bf7885dba92daa85
I20260926 19:27:28.524909 6604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:28.535282 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:28.523492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42620 (local address 127.1.39.62:33865)
0926 19:27:28.524681 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:28.524692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:28.524735 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:27:28.526502 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:28.526523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:28.526956 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:28.527289 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:28.527880 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.529160 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.531000 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.531630 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.531752 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:28.532949 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:28.532956 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:28.532968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:28.533598 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:28.533678 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:28.533700 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:28.533791 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:28.534014 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 19:27:28.534360 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":389,"threads_started":1}
I20260926 19:27:28.536648 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c: Bootstrap starting.
I20260926 19:27:28.538910 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:27:28.540056 6604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:28.541194 6604 log.cc:824] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:28.544085 6604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c: No bootstrap required, opened a new log
I20260926 19:27:28.551819 6604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } }
I20260926 19:27:28.552168 6604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:28.552299 6604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3035f81b89b64da292883649c2cdfb9c, State: Initialized, Role: FOLLOWER
I20260926 19:27:28.552903 6604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } }
I20260926 19:27:28.553128 6604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:28.553264 6604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:28.553464 6604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:28.555612 6604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } }
I20260926 19:27:28.556152 6604 leader_election.cc:302] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3035f81b89b64da292883649c2cdfb9c; no voters:
I20260926 19:27:28.556718 6604 leader_election.cc:288] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:28.556876 6609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:28.557608 6609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [term 1 LEADER]: Becoming Leader. State: Replica: 3035f81b89b64da292883649c2cdfb9c, State: Running, Role: LEADER
I20260926 19:27:28.558168 6609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } }
I20260926 19:27:28.558743 6604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:28.561125 6610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3035f81b89b64da292883649c2cdfb9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } } }
I20260926 19:27:28.561381 6611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3035f81b89b64da292883649c2cdfb9c. Latest consensus state: current_term: 1 leader_uuid: "3035f81b89b64da292883649c2cdfb9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3035f81b89b64da292883649c2cdfb9c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33865 } } }
I20260926 19:27:28.561638 6610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:28.561702 6611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:28.563244 6618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:28.567835 6618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:28.575627 6618 catalog_manager.cc:1347] Generated new cluster ID: 6bd3ed8bfcae43de875bd364a4a00538
I20260926 19:27:28.575749 6618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:28.609834 6618 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:28.611179 6618 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:28.619859 6618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3035f81b89b64da292883649c2cdfb9c: Generated new TSK 0
I20260926 19:27:28.620736 6618 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:27:28.647082 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:28.638510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.1.39.62:33865)
0926 19:27:28.638681 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:28.638686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:28.639172 (+ 486us) server_negotiation.cc:407] Connection header received
0926 19:27:28.639421 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:28.639425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:28.639492 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:28.639651 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:27:28.640221 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.641415 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.642725 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.643417 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.643548 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:28.644042 (+ 494us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:27:28.646439 (+ 2397us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36235/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 19:27:28.646784 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36235/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 19:27:28.647217 6607 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36235/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 19:27:28.648944 6633 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:28.638497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33865 (local address 127.0.0.1:42622)
0926 19:27:28.639071 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:28.639086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:28.639242 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:28.639661 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:28.639666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:28.639678 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:27:28.640041 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:28.640057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.641562 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.641566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:28.642595 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:28.642605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:28.643547 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:28.643621 (+ 74us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:28.643744 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:28.643774 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:27:28.646680 (+ 2906us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36235/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 19:27:28.648475 (+ 1795us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.39.62:33865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36235/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1447,"thread_start_us":332,"threads_started":1}
I20260926 19:27:28.650614 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6538
I20260926 19:27:28.651798 6599 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 19:27:28.659387 6634 webserver.cc:533] Webserver: sslize failed:
E20260926 19:27:28.659515 6634 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 19:27:28.743965 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6538
2026-09-26T19:27:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (736 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T19:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:28Z Disabled control of system clock
I20260926 19:27:28.826432 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38521
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38521
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/testchainca.cert with env {}
W20260926 19:27:29.147553 6639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:29.147823 6639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:29.147914 6639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:29.155122 6639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:29.155220 6639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:29.155256 6639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:29.155314 6639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:29.155357 6639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 19:27:29.155417 6639 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 19:27:29.155483 6639 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 19:27:29.165118 6639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:29.166353 6639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:29.167593 6639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:29.179380 6645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:29.179523 6644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:29.179450 6647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:29.180392 6639 server_base.cc:1034] running on GCE node
I20260926 19:27:29.181170 6639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:29.182529 6639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:29.183730 6639 hybrid_clock.cc:648] HybridClock initialized: now 1790450849183691 us; error 30 us; skew 500 ppm
I20260926 19:27:29.184023 6639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:27:29.186201 6639 webserver.cc:460] Webserver started at http://127.1.39.62:35335/ using document root <none> and password file <none>
I20260926 19:27:29.186837 6639 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:29.186924 6639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:29.187177 6639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:29.189944 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "e6dc0a05fd2d46b9888095e1a301c986"
format_stamp: "Formatted at 2026-09-26 19:27:29 on dist-test-slave-9wb0"
server_key: "afd4ba0f10b592ece90cc0c33cd1e78d"
server_key_iv: "9bc64c819ef2223abc4a39f0c52252ef"
server_key_version: "encryptionkey@0"
I20260926 19:27:29.190675 6639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "e6dc0a05fd2d46b9888095e1a301c986"
format_stamp: "Formatted at 2026-09-26 19:27:29 on dist-test-slave-9wb0"
server_key: "afd4ba0f10b592ece90cc0c33cd1e78d"
server_key_iv: "9bc64c819ef2223abc4a39f0c52252ef"
server_key_version: "encryptionkey@0"
I20260926 19:27:29.195446 6639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:27:29.198837 6653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:29.200168 6639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:27:29.200349 6639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "e6dc0a05fd2d46b9888095e1a301c986"
format_stamp: "Formatted at 2026-09-26 19:27:29 on dist-test-slave-9wb0"
server_key: "afd4ba0f10b592ece90cc0c33cd1e78d"
server_key_iv: "9bc64c819ef2223abc4a39f0c52252ef"
server_key_version: "encryptionkey@0"
I20260926 19:27:29.200533 6639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:29.223506 6639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:29.224325 6639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:29.224622 6639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:29.244829 6639 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:38521
I20260926 19:27:29.244837 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:38521 every 8 connection(s)
I20260926 19:27:29.246168 6639 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:29.249801 6705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:29.254915 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6639
I20260926 19:27:29.255144 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:29.255829 1180 external_mini_cluster.cc:1428] Setting key 85fe90253a9fb8c6c326eae916fbcda7
I20260926 19:27:29.260509 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986: Bootstrap starting.
I20260926 19:27:29.263550 6705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:29.264490 6705 log.cc:824] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:29.267424 6705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986: No bootstrap required, opened a new log
I20260926 19:27:29.269043 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:29.257747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.1.39.62:38521)
0926 19:27:29.258789 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:29.258800 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:29.258849 (+ 49us) server_negotiation.cc:407] Connection header received
0926 19:27:29.260443 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:29.260467 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:29.261002 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:29.261406 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:29.261953 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.263203 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.264548 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.265374 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.265600 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:29.266807 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:29.266814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:29.266827 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:27:29.267351 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:29.267428 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:27:29.267448 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:29.267590 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:29.267808 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 19:27:29.268140 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":359,"threads_started":1}
I20260926 19:27:29.272715 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:27:29.275277 6705 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } }
I20260926 19:27:29.275651 6705 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:29.275758 6705 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6dc0a05fd2d46b9888095e1a301c986, State: Initialized, Role: FOLLOWER
I20260926 19:27:29.276286 6705 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } }
I20260926 19:27:29.276522 6705 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:29.276687 6705 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:29.276865 6705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:29.279024 6705 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } }
I20260926 19:27:29.279516 6705 leader_election.cc:302] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6dc0a05fd2d46b9888095e1a301c986; no voters:
I20260926 19:27:29.280220 6705 leader_election.cc:288] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:29.280339 6710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:29.281044 6710 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [term 1 LEADER]: Becoming Leader. State: Replica: e6dc0a05fd2d46b9888095e1a301c986, State: Running, Role: LEADER
I20260926 19:27:29.281620 6710 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } }
I20260926 19:27:29.282524 6705 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:29.284730 6711 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6dc0a05fd2d46b9888095e1a301c986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } } }
I20260926 19:27:29.285159 6711 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:29.285326 6712 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6dc0a05fd2d46b9888095e1a301c986. Latest consensus state: current_term: 1 leader_uuid: "e6dc0a05fd2d46b9888095e1a301c986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6dc0a05fd2d46b9888095e1a301c986" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38521 } } }
I20260926 19:27:29.285769 6712 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:29.289623 6716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:29.294945 6716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:29.303728 6716 catalog_manager.cc:1347] Generated new cluster ID: 5d610e7d293547e48bcef91c6ea2741b
I20260926 19:27:29.303867 6716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:29.314616 6716 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:29.316473 6716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:29.324805 6716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6dc0a05fd2d46b9888095e1a301c986: Generated new TSK 0
I20260926 19:27:29.325708 6716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:27:29.775847 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:29.340485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.1.39.62:38521)
0926 19:27:29.340669 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:29.340675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:29.341186 (+ 511us) server_negotiation.cc:407] Connection header received
0926 19:27:29.341414 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:29.341420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:29.341520 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:29.341682 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0926 19:27:29.342121 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.343204 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.344521 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.345189 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.345336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:29.345811 (+ 475us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 19:27:29.775284 (+429473us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 19:27:29.775627 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51150: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 19:27:29.775955 6708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51150: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 19:27:29.776230 6734 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:29.340486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38521 (local address 127.0.0.1:51150)
0926 19:27:29.341081 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:29.341104 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:29.341215 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:29.341714 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:29.341718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:29.341727 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 19:27:29.341977 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:29.341985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.343337 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.343340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:29.344389 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:29.344399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:29.345354 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:29.345396 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:29.345519 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:29.345549 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 19:27:29.775594 (+430045us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 19:27:29.775841 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.39.62:38521: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
I20260926 19:27:29.777415 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6639
I20260926 19:27:29.778527 6700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:29.881294 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6639
2026-09-26T19:27:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1119 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (414 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (407 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:30 dist-test-slave-9wb0 krb5kdc[6749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:30 dist-test-slave-9wb0 krb5kdc[6749](info): set up 2 sockets
Sep 26 19:27:30 dist-test-slave-9wb0 krb5kdc[6749](info): commencing operation
krb5kdc: starting...
W20260926 19:27:31.814059 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:31 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:32.029066 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33857
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:32.362848 6762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:32.363179 6762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:32.363332 6762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:32.370573 6762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:32.370723 6762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:32.370822 6762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:32.370921 6762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:32.370995 6762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:32.380785 6762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:32.382122 6762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:32.383392 6762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:32.394824 6767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:32.394915 6770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:32.395124 6768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:32.396919 6762 server_base.cc:1034] running on GCE node
I20260926 19:27:32.397792 6762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:32.399161 6762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:32.400384 6762 hybrid_clock.cc:648] HybridClock initialized: now 1790450852400342 us; error 39 us; skew 500 ppm
I20260926 19:27:32.400722 6762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:32 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:32.403720 6762 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:32.405153 6762 webserver.cc:460] Webserver started at http://127.1.39.62:42751/ using document root <none> and password file <none>
I20260926 19:27:32.405850 6762 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:32.405947 6762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:32.406210 6762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:32.408998 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "707a35f548f746f0b63e46eaeddcec6b"
format_stamp: "Formatted at 2026-09-26 19:27:32 on dist-test-slave-9wb0"
server_key: "53e9d64ecb7f3b70262307265ef932f2"
server_key_iv: "ef28672b17d2a59bdfd2581fa4517387"
server_key_version: "encryptionkey@0"
I20260926 19:27:32.409787 6762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "707a35f548f746f0b63e46eaeddcec6b"
format_stamp: "Formatted at 2026-09-26 19:27:32 on dist-test-slave-9wb0"
server_key: "53e9d64ecb7f3b70262307265ef932f2"
server_key_iv: "ef28672b17d2a59bdfd2581fa4517387"
server_key_version: "encryptionkey@0"
I20260926 19:27:32.414659 6762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 19:27:32.417927 6777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:32.419170 6762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:32.419344 6762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "707a35f548f746f0b63e46eaeddcec6b"
format_stamp: "Formatted at 2026-09-26 19:27:32 on dist-test-slave-9wb0"
server_key: "53e9d64ecb7f3b70262307265ef932f2"
server_key_iv: "ef28672b17d2a59bdfd2581fa4517387"
server_key_version: "encryptionkey@0"
I20260926 19:27:32.419529 6762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:32.453802 6762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:32.457701 6762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:32.457984 6762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:32.475934 6762 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33857
I20260926 19:27:32.475965 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33857 every 8 connection(s)
I20260926 19:27:32.477272 6762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:32.480160 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6762
I20260926 19:27:32.480391 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:32.481213 1180 external_mini_cluster.cc:1428] Setting key 79c3fc64e155115a0c092d0c74d318d8
I20260926 19:27:32.481525 6829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:32.498381 6829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b: Bootstrap starting.
Sep 26 19:27:32 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:32.502067 6829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:32.503451 6829 log.cc:824] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:32.506981 6829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b: No bootstrap required, opened a new log
I20260926 19:27:32.508816 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:32.483283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.1.39.62:33857)
0926 19:27:32.484062 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:32.484073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:32.484119 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:27:32.487898 (+ 3779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:32.487931 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:32.488518 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:32.489000 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:32.491881 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:32.493767 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:32.495059 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:32.496156 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:32.496345 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:32.500202 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:32.500218 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:32.500237 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:32.500286 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:32.504089 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:32.504947 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:32.504955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:32.504963 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:32.505170 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:32.505700 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:32.505704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:32.505708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:32.506400 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:32.506668 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:32.507053 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 19:27:32.507518 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":374,"threads_started":1}
I20260926 19:27:32.515419 6829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } }
I20260926 19:27:32.515750 6829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:32.515826 6829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707a35f548f746f0b63e46eaeddcec6b, State: Initialized, Role: FOLLOWER
I20260926 19:27:32.516312 6829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } }
I20260926 19:27:32.516497 6829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:32.516592 6829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:32.516713 6829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:32.519052 6829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } }
I20260926 19:27:32.519474 6829 leader_election.cc:302] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 707a35f548f746f0b63e46eaeddcec6b; no voters:
I20260926 19:27:32.520008 6829 leader_election.cc:288] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:32.520124 6834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:32.520869 6834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [term 1 LEADER]: Becoming Leader. State: Replica: 707a35f548f746f0b63e46eaeddcec6b, State: Running, Role: LEADER
I20260926 19:27:32.521391 6834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } }
I20260926 19:27:32.522240 6829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:32.524592 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "707a35f548f746f0b63e46eaeddcec6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } } }
I20260926 19:27:32.525063 6836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 707a35f548f746f0b63e46eaeddcec6b. Latest consensus state: current_term: 1 leader_uuid: "707a35f548f746f0b63e46eaeddcec6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "707a35f548f746f0b63e46eaeddcec6b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33857 } } }
I20260926 19:27:32.525748 6836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:32.525748 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:32.529785 6843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:32.535130 6843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:32.543876 6843 catalog_manager.cc:1347] Generated new cluster ID: b7ea19f2ec514105970ccaa18e71531c
I20260926 19:27:32.543987 6843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:32.555845 6843 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:32.557760 6843 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:32.573015 6843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 707a35f548f746f0b63e46eaeddcec6b: Generated new TSK 0
I20260926 19:27:32.573976 6843 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:32.628438 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:32.965758 6855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:32.966039 6855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:32.966136 6855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:32.973428 6855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:32.973544 6855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:32.973659 6855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:32.983174 6855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:32.984386 6855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:32.985714 6855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:32.998631 6863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:32.998595 6861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:32.998674 6860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:32.999248 6855 server_base.cc:1034] running on GCE node
I20260926 19:27:32.999754 6855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:33.000646 6855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:33.001843 6855 hybrid_clock.cc:648] HybridClock initialized: now 1790450853001810 us; error 33 us; skew 500 ppm
I20260926 19:27:33.002102 6855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:32 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:33.004982 6855 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:33.006457 6855 webserver.cc:460] Webserver started at http://127.1.39.1:40559/ using document root <none> and password file <none>
I20260926 19:27:33.007102 6855 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:33.007230 6855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:33.007534 6855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:33.010382 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "38f53f2404e649e299628f39e6f5cbd8"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "3fb3db81cd2ced948b2899e875acbaa6"
server_key_iv: "44622332262fde47571f97bdb1914811"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.011169 6855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "38f53f2404e649e299628f39e6f5cbd8"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "3fb3db81cd2ced948b2899e875acbaa6"
server_key_iv: "44622332262fde47571f97bdb1914811"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.016134 6855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:27:33.019446 6870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.020725 6855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:27:33.020889 6855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "38f53f2404e649e299628f39e6f5cbd8"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "3fb3db81cd2ced948b2899e875acbaa6"
server_key_iv: "44622332262fde47571f97bdb1914811"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.021080 6855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:33.042407 6855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:33.046459 6855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:33.046756 6855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:33.047703 6855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:33.049000 6855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:33.049104 6855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.049247 6855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:33.049346 6855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.082024 6855 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:32935
I20260926 19:27:33.082062 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:32935 every 8 connection(s)
I20260926 19:27:33.083266 6855 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:33.092497 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6855
I20260926 19:27:33.092710 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:33.093530 1180 external_mini_cluster.cc:1428] Setting key 1599f1abe706c7bea102b3c25f86908c
Sep 26 19:27:33 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450852, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:33.103559 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:33.085829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:43967 (local address 127.1.39.62:33857)
0926 19:27:33.086263 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:33.086268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:33.087147 (+ 879us) server_negotiation.cc:407] Connection header received
0926 19:27:33.088894 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:33.088899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:33.088987 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:33.089105 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:33.090649 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.091370 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.093537 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.093991 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.094101 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:33.098066 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:33.098072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:33.098075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:33.098105 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:33.100562 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:33.101287 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:33.101329 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:33.101333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:33.101456 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:33.101873 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:33.101878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:33.101881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:33.102134 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:33.102283 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:33.102961 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 19:27:33.103177 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":240,"threads_started":1}
I20260926 19:27:33.104532 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:33.086276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33857 (local address 127.1.39.1:43967)
0926 19:27:33.086975 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:33.087011 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:33.088627 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:33.089396 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:33.089404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:33.089907 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:33.090474 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:33.090491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.091519 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.091531 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:33.093386 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:33.093403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.094256 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.094266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:33.094514 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:33.095272 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:33.095311 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:33.097859 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:33.100708 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:33.100717 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:33.100735 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:33.101161 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:33.101563 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:33.101569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:33.101576 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:33.101756 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:33.102260 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:33.102268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:33.102601 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 19:27:33.102858 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 19:27:33.103407 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":282,"threads_started":1}
I20260926 19:27:33.106086 6983 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33857
I20260926 19:27:33.106396 6983 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:33.107285 6983 heartbeater.cc:511] Master 127.1.39.62:33857 requested a full tablet report, sending...
I20260926 19:27:33.109644 6794 ts_manager.cc:194] Registered new tserver with Master: 38f53f2404e649e299628f39e6f5cbd8 (127.1.39.1:32935)
I20260926 19:27:33.111799 6794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:43967
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:33.143996 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:33.464107 6989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:33.464391 6989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:33.464560 6989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:33.471864 6989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:33.471977 6989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:33.472157 6989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:33.481882 6989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:33.483086 6989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:33.484402 6989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:33.501247 6994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:33.501267 6995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:33.501247 6997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:33.502498 6989 server_base.cc:1034] running on GCE node
I20260926 19:27:33.503186 6989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:33.504135 6989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:33.505404 6989 hybrid_clock.cc:648] HybridClock initialized: now 1790450853505352 us; error 32 us; skew 500 ppm
I20260926 19:27:33.505741 6989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:33 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:33.508795 6989 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:33.510232 6989 webserver.cc:460] Webserver started at http://127.1.39.2:37415/ using document root <none> and password file <none>
I20260926 19:27:33.510932 6989 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:33.511080 6989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:33.511379 6989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:33.514469 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "84944b5b2a5d4f5eb9c2ecbf8a941a7f"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "047d0b7f52f9debf03cc8d317aae1f57"
server_key_iv: "109f50e4b7936dda883cb653aff62517"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.515282 6989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "84944b5b2a5d4f5eb9c2ecbf8a941a7f"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "047d0b7f52f9debf03cc8d317aae1f57"
server_key_iv: "109f50e4b7936dda883cb653aff62517"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.520313 6989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:27:33.524606 7004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.526690 6989 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:27:33.526872 6989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "84944b5b2a5d4f5eb9c2ecbf8a941a7f"
format_stamp: "Formatted at 2026-09-26 19:27:33 on dist-test-slave-9wb0"
server_key: "047d0b7f52f9debf03cc8d317aae1f57"
server_key_iv: "109f50e4b7936dda883cb653aff62517"
server_key_version: "encryptionkey@0"
I20260926 19:27:33.527125 6989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:33.542692 6989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:33.546602 6989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:33.546887 6989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:33.547845 6989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:33.549150 6989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:33.549254 6989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.549436 6989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:33.549525 6989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:33.582746 6989 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:45309
I20260926 19:27:33.582783 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:45309 every 8 connection(s)
I20260926 19:27:33.584118 6989 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:33.594429 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 6989
I20260926 19:27:33.594636 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:33.595188 1180 external_mini_cluster.cc:1428] Setting key 2e57215578d3f49529e6a71b5084357d
Sep 26 19:27:33 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450853, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:33.605542 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:33.586652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:60301 (local address 127.1.39.62:33857)
0926 19:27:33.586837 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:33.586842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:33.588007 (+ 1165us) server_negotiation.cc:407] Connection header received
0926 19:27:33.589836 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:33.589842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:33.589933 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:33.590053 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:33.591653 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.592356 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.594081 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.594759 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.594898 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:33.598511 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:33.598520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:33.598528 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:33.598574 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:33.602181 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:33.602922 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:33.602930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:33.602936 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:33.603100 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:33.603553 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:33.603561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:33.603567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:33.603927 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:33.604088 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:33.604852 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 19:27:33.605121 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:27:33.606627 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:33.587026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33857 (local address 127.1.39.2:60301)
0926 19:27:33.587831 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:33.587872 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:33.589573 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:33.590290 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:33.590299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:33.590811 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:33.591456 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:33.591481 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.592491 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.592494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:33.593918 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:33.593930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:33.594959 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:33.594964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:33.595162 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:33.595943 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:33.595976 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:33.598279 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:33.602330 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:33.602337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:33.602356 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:33.602783 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:33.603216 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:33.603219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:33.603222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:33.603405 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:33.604083 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:33.604090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:33.604441 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 19:27:33.604748 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 19:27:33.605310 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
I20260926 19:27:33.608168 7117 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33857
I20260926 19:27:33.608494 7117 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:33.609409 7117 heartbeater.cc:511] Master 127.1.39.62:33857 requested a full tablet report, sending...
I20260926 19:27:33.611011 6794 ts_manager.cc:194] Registered new tserver with Master: 84944b5b2a5d4f5eb9c2ecbf8a941a7f (127.1.39.2:45309)
I20260926 19:27:33.611876 6794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:60301
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:33.644860 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:33.971290 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:33.971626 7122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:33.971805 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:33.978971 7122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:33.979131 7122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:33.979324 7122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:33.988875 7122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:33.990198 7122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:33.991523 7122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:34.004271 7127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:34.004355 7128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:34.004252 7130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:34.006718 7122 server_base.cc:1034] running on GCE node
I20260926 19:27:34.007431 7122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:34.008606 7122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:34.009882 7122 hybrid_clock.cc:648] HybridClock initialized: now 1790450854009814 us; error 60 us; skew 500 ppm
I20260926 19:27:34.010375 7122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.013868 7122 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:34.015461 7122 webserver.cc:460] Webserver started at http://127.1.39.3:35417/ using document root <none> and password file <none>
I20260926 19:27:34.016223 7122 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:34.016361 7122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:34.016722 7122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:34.019945 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "e01dd418fabd4fb99e954443679b2f0a"
format_stamp: "Formatted at 2026-09-26 19:27:34 on dist-test-slave-9wb0"
server_key: "acf8874c3008b668cffdce15f2538889"
server_key_iv: "7994e3d431ffca4268da9b29e6643c7b"
server_key_version: "encryptionkey@0"
I20260926 19:27:34.020771 7122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "e01dd418fabd4fb99e954443679b2f0a"
format_stamp: "Formatted at 2026-09-26 19:27:34 on dist-test-slave-9wb0"
server_key: "acf8874c3008b668cffdce15f2538889"
server_key_iv: "7994e3d431ffca4268da9b29e6643c7b"
server_key_version: "encryptionkey@0"
I20260926 19:27:34.026777 7122 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 19:27:34.030224 7137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:34.031646 7122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:34.031812 7122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "e01dd418fabd4fb99e954443679b2f0a"
format_stamp: "Formatted at 2026-09-26 19:27:34 on dist-test-slave-9wb0"
server_key: "acf8874c3008b668cffdce15f2538889"
server_key_iv: "7994e3d431ffca4268da9b29e6643c7b"
server_key_version: "encryptionkey@0"
I20260926 19:27:34.032001 7122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:34.057776 7122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:34.063115 7122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:34.063498 7122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:34.064802 7122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:34.066555 7122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:34.066648 7122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:34.066782 7122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:34.066851 7122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:34.101070 7122 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37911
I20260926 19:27:34.101118 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37911 every 8 connection(s)
I20260926 19:27:34.102531 7122 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:34.105400 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7122
I20260926 19:27:34.105572 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:34.106014 1180 external_mini_cluster.cc:1428] Setting key 86d2ad661a229c42e5d7e43fd879a2a3
I20260926 19:27:34.115407 6983 heartbeater.cc:503] Master 127.1.39.62:33857 was elected leader, sending a full tablet report...
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:34.123826 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.105160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:50105 (local address 127.1.39.62:33857)
0926 19:27:34.105402 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.105409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.106790 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 19:27:34.109059 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.109084 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.109175 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.109274 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.110768 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.111614 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.113126 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.113629 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.113732 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.117439 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.117443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.117446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.117487 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.120157 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.121152 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.121157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.121159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.121388 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.121980 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.121983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.121986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.122325 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.122458 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.123296 (+ 838us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.123527 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:27:34.124854 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.105784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33857 (local address 127.1.39.3:50105)
0926 19:27:34.106639 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:34.106685 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:34.108781 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:34.109553 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:34.109562 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:34.110048 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:34.110608 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:34.110630 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.111752 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.111756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:34.112963 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:34.112974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.113812 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.113817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:34.114018 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.114630 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:34.114651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:34.117183 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:34.120292 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:34.120302 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:34.120321 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:34.121014 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:34.121514 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:34.121518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:34.121520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:34.121829 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:34.122445 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:34.122452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:34.122885 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 19:27:34.123143 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 19:27:34.123620 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":342,"threads_started":1}
I20260926 19:27:34.126205 7250 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33857
I20260926 19:27:34.126518 7250 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:34.127239 7250 heartbeater.cc:511] Master 127.1.39.62:33857 requested a full tablet report, sending...
I20260926 19:27:34.128649 6794 ts_manager.cc:194] Registered new tserver with Master: e01dd418fabd4fb99e954443679b2f0a (127.1.39.3:37911)
I20260926 19:27:34.129594 6794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:50105
I20260926 19:27:34.133929 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/test-admin.keytab.
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.160667 1180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 19:27:34.165967 7257 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:34.163334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.1.39.1:32935)
0926 19:27:34.164096 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.164103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.164128 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:27:34.164185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.164192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.164397 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.164584 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.165489 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32942: BlockingRecv error: recv got EOF from 127.0.0.1:32942 (error 108)
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":402,"threads_started":1}
W20260926 19:27:34.166023 1180 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc' of type FILE
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.167557 1180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.168869 1180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:34.183873 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.170481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.1.39.1:32935)
0926 19:27:34.170689 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.170695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.171011 (+ 316us) server_negotiation.cc:407] Connection header received
0926 19:27:34.171219 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.171223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.171276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.171381 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.172357 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.173282 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.174526 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.175075 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.175207 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.178642 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.178652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.178666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.178696 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.181265 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.181923 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.181931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.181940 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.182052 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.182388 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.182394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.182397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.182727 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.183023 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.183340 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.183595 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 19:27:34.189285 7257 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:34.187864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.1.39.1:32935)
0926 19:27:34.188058 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.188069 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.188152 (+ 83us) server_negotiation.cc:407] Connection header received
0926 19:27:34.188263 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.188267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.188319 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.188395 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.189006 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32962: BlockingRecv error: recv got EOF from 127.0.0.1:32962 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 19:27:34.189611 1180 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc)
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.191092 1180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.192307 1180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:34.206337 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.193740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.1.39.1:32935)
0926 19:27:34.193928 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.193934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.194128 (+ 194us) server_negotiation.cc:407] Connection header received
0926 19:27:34.194228 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.194234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.194302 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.194389 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.195233 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.195911 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.197039 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.197544 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.197640 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.201211 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.201217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.201221 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.201263 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.203871 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.204524 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.204529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.204531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.204669 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.205024 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.205029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.205031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.205371 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.205615 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.205911 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.206095 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 19:27:34.211143 7257 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:34.209105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32992 (local address 127.1.39.1:32935)
0926 19:27:34.209383 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.209391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.209698 (+ 307us) server_negotiation.cc:407] Connection header received
0926 19:27:34.209886 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.209899 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.209984 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.210094 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.210861 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32992: BlockingRecv error: recv got EOF from 127.0.0.1:32992 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 19:27:34.211295 1180 init.cc:244] Error while opening credential cache '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790450801920901-1180-0/krb5kdc/krb5cc)
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.212800 1180 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:34.214048 1180 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:34.227819 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.215211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.1.39.1:32935)
0926 19:27:34.215474 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.215481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.215756 (+ 275us) server_negotiation.cc:407] Connection header received
0926 19:27:34.215868 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.215871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.215929 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.216011 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.216927 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.217789 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.218973 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.219379 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.219479 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.222696 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.222701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.222704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.222731 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.225584 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.226240 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.226244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.226246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.226365 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.226647 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.226650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.226652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.226933 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.227158 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.227320 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.227504 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:27:34.229259 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6855
I20260926 19:27:34.241531 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.229974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.1.39.1:32935)
0926 19:27:34.230134 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.230151 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.230465 (+ 314us) server_negotiation.cc:407] Connection header received
0926 19:27:34.230632 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.230644 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.230693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.230815 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.231762 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.232400 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.233719 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.234107 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.234231 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.236336 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.236341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.236345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.236375 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.238859 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.239520 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.239528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.239531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.239675 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.240052 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.240059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.240063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.240306 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.240599 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.240932 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.241135 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:27:34.242064 6978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:34.330856 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6855
I20260926 19:27:34.348237 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6989
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:34.364874 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.349089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.1.39.2:45309)
0926 19:27:34.349646 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.349654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.349678 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:27:34.349735 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.349741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.349947 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.350080 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.351013 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.352366 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.353869 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.354554 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.354661 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.358368 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.358378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.358392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.358423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.362164 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.362804 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.362811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.362813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.362931 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.363343 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.363346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.363348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.363715 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.363948 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.364283 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.364516 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":310,"threads_started":1}
I20260926 19:27:34.366333 7112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:34.448892 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6989
I20260926 19:27:34.464920 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7122
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[6749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:34.480341 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:34.465927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.1.39.3:37911)
0926 19:27:34.466536 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:34.466543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:34.466570 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:27:34.466627 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:34.466632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:34.466833 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:34.467004 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:34.468083 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.469103 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.470379 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:34.470954 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:34.471096 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:34.474911 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:34.474922 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:34.474936 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:34.474967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:34.477733 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.478391 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.478397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.478399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.478510 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:34.478879 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:34.478894 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:34.478901 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:34.479212 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:34.479517 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:34.479803 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:27:34.480050 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":304,"threads_started":1}
I20260926 19:27:34.481808 7245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:34.573400 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7122
I20260926 19:27:34.589531 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 6762
I20260926 19:27:34.590739 6824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:34.691329 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 6762
2026-09-26T19:27:34Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3992 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[7277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[7277](info): set up 2 sockets
Sep 26 19:27:34 dist-test-slave-9wb0 krb5kdc[7277](info): commencing operation
krb5kdc: starting...
W20260926 19:27:35.790129 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:35 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab.
Entry for principal oryx/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab.
I20260926 19:27:36.013793 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33725
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab
--principal=oryx/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:36.336308 7290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:36.336653 7290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:36.336804 7290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:36.343922 7290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:36.344060 7290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:36.344169 7290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:36.344269 7290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:36.344344 7290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:36.353940 7290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab
--principal=oryx/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:36.355222 7290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:36.356477 7290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:36.368010 7296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:36.367976 7298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:36.368244 7295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:36.369072 7290 server_base.cc:1034] running on GCE node
I20260926 19:27:36.369882 7290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:36.371204 7290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:36.372409 7290 hybrid_clock.cc:648] HybridClock initialized: now 1790450856372363 us; error 27 us; skew 500 ppm
I20260926 19:27:36.372750 7290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:36 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:36.380934 7290 init.cc:377] Logged in from keytab as oryx/127.1.39.62@KRBTEST.COM (short username oryx)
I20260926 19:27:36.382479 7290 webserver.cc:460] Webserver started at http://127.1.39.62:39107/ using document root <none> and password file <none>
I20260926 19:27:36.383225 7290 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:36.383366 7290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:36.383656 7290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:36.386602 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "d0b3c26acf0547b9a575729235755a20"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "044e8302c071ad7069971c3249abf743"
server_key_iv: "d54cdd731f8b3c19876a1870f48e486a"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.387332 7290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "d0b3c26acf0547b9a575729235755a20"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "044e8302c071ad7069971c3249abf743"
server_key_iv: "d54cdd731f8b3c19876a1870f48e486a"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.392168 7290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 19:27:36.395416 7305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:36.396610 7290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 19:27:36.396782 7290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "d0b3c26acf0547b9a575729235755a20"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "044e8302c071ad7069971c3249abf743"
server_key_iv: "d54cdd731f8b3c19876a1870f48e486a"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.396965 7290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:36.420727 7290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:36.424957 7290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:36.425264 7290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:36.444335 7290 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33725
I20260926 19:27:36.444346 7356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33725 every 8 connection(s)
I20260926 19:27:36.445734 7290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:36.449220 7357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:36.454574 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7290
I20260926 19:27:36.454778 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:36.455619 1180 external_mini_cluster.cc:1428] Setting key 2e64a928ea5b875a43bd36186381dd69
I20260926 19:27:36.460548 7357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Bootstrap starting.
I20260926 19:27:36.463512 7357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:36.464520 7357 log.cc:824] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:36.467617 7357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: No bootstrap required, opened a new log
Sep 26 19:27:36 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:36.478123 7357 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } }
I20260926 19:27:36.478562 7357 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:36.478595 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:36.457809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52254 (local address 127.1.39.62:33725)
0926 19:27:36.458608 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:36.458623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:36.458664 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:27:36.460283 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:36.460304 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:36.460726 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:36.461103 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:36.462514 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:36.463817 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:36.465133 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:36.466076 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:36.466266 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:36.470229 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:36.470241 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:36.470267 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:36.470314 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:36.474317 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:36.474949 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:36.474957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:36.474965 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:36.475166 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:36.475506 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:36.475511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:36.475515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:36.476212 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:36.476536 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:36.476816 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 19:27:36.477236 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":379,"threads_started":1}
I20260926 19:27:36.478696 7357 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b3c26acf0547b9a575729235755a20, State: Initialized, Role: FOLLOWER
I20260926 19:27:36.479311 7357 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } }
I20260926 19:27:36.479568 7357 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:36.479708 7357 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:36.479875 7357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:36.482342 7357 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } }
I20260926 19:27:36.482805 7357 leader_election.cc:302] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0b3c26acf0547b9a575729235755a20; no voters:
I20260926 19:27:36.483371 7357 leader_election.cc:288] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:36.483489 7362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:36.486431 7357 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:36.487440 7362 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 LEADER]: Becoming Leader. State: Replica: d0b3c26acf0547b9a575729235755a20, State: Running, Role: LEADER
I20260926 19:27:36.488118 7362 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } }
I20260926 19:27:36.493561 7363 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } }
I20260926 19:27:36.493582 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b3c26acf0547b9a575729235755a20. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } }
I20260926 19:27:36.494023 7363 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:36.494022 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:36.502112 7372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:36.506738 7372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:36.514974 7372 catalog_manager.cc:1347] Generated new cluster ID: 0293c0558520431e87dd0a0ffc992310
I20260926 19:27:36.515106 7372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:36.540190 7372 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:36.541517 7372 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:36.549242 7372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Generated new TSK 0
I20260926 19:27:36.550040 7372 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
I20260926 19:27:36.602521 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:36.926004 7383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:36.926291 7383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:36.926378 7383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:36.933743 7383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:36.933845 7383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:36.933955 7383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:36.943428 7383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:36.944660 7383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:36.945971 7383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:36.959290 7389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:36.959317 7391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:36.959380 7388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:36.960388 7383 server_base.cc:1034] running on GCE node
I20260926 19:27:36.961015 7383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:36.961961 7383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:36.963165 7383 hybrid_clock.cc:648] HybridClock initialized: now 1790450856963123 us; error 37 us; skew 500 ppm
I20260926 19:27:36.963483 7383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:36 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:36.966711 7383 init.cc:377] Logged in from keytab as oryx/127.1.39.1@KRBTEST.COM (short username oryx)
I20260926 19:27:36.968286 7383 webserver.cc:460] Webserver started at http://127.1.39.1:42053/ using document root <none> and password file <none>
I20260926 19:27:36.968956 7383 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:36.969054 7383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:36.969354 7383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:36.972312 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "827c49b220cf4c4a98ae0a3c38f06420"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "44e0a1c0ff6149484558c265e05f8dab"
server_key_iv: "55cf394afa6a2ebea6803297001868c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.973062 7383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "827c49b220cf4c4a98ae0a3c38f06420"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "44e0a1c0ff6149484558c265e05f8dab"
server_key_iv: "55cf394afa6a2ebea6803297001868c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.978358 7383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:27:36.981716 7398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:36.982964 7383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:27:36.983160 7383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "827c49b220cf4c4a98ae0a3c38f06420"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "44e0a1c0ff6149484558c265e05f8dab"
server_key_iv: "55cf394afa6a2ebea6803297001868c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:36.983373 7383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:37.004730 7383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:37.008653 7383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:37.008931 7383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:37.009922 7383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:37.011176 7383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:37.011283 7383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.011417 7383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:37.011473 7383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.043638 7383 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33623
I20260926 19:27:37.043684 7510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33623 every 8 connection(s)
I20260926 19:27:37.044866 7383 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:37.052350 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7383
I20260926 19:27:37.052554 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:37.053162 1180 external_mini_cluster.cc:1428] Setting key 6eca8bead54b63626f72e84fca75a781
Sep 26 19:27:37 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:37.064339 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:37.047345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:58507 (local address 127.1.39.62:33725)
0926 19:27:37.047762 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:37.047766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:37.048724 (+ 958us) server_negotiation.cc:407] Connection header received
0926 19:27:37.050608 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:37.050613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:37.050694 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:37.050807 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:37.052272 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.053096 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.054844 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.055332 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.055464 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:37.058990 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:37.058995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:37.058998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:37.059027 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:37.061503 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:37.062182 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:37.062194 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:37.062196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:37.062299 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:37.062711 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:37.062716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:37.062719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:37.062960 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:37.063129 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:37.063786 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 19:27:37.063988 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":222,"threads_started":1}
I20260926 19:27:37.065249 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:37.047914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:58507)
0926 19:27:37.048570 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:37.048611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:37.050332 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:37.051053 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:37.051061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:37.051580 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:37.052108 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:37.052128 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.053248 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.053272 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:37.054695 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:37.054706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.055544 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.055552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:37.055775 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:37.056442 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:37.056464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:37.058790 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:37.061641 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:37.061649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:37.061665 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:37.062060 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:37.062406 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:37.062411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:37.062415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:37.062596 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:37.063101 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:37.063109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:37.063419 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 19:27:37.063673 (+ 254us) client_negotiation.cc:240] Negotiation successful
0926 19:27:37.064159 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":283,"threads_started":1}
I20260926 19:27:37.067690 7511 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:37.067998 7511 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:37.068687 7511 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
I20260926 19:27:37.071096 7322 ts_manager.cc:194] Registered new tserver with Master: 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623)
I20260926 19:27:37.073352 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:58507
WARNING: no policy specified for oryx/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
I20260926 19:27:37.101972 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:37.419736 7517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:37.420045 7517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:37.420226 7517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:37.427205 7517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:37.427357 7517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:37.427552 7517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:37.437214 7517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:37.438915 7517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:37.440375 7517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:37.457994 7523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:37.458400 7525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:37.457974 7522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:37.459630 7517 server_base.cc:1034] running on GCE node
I20260926 19:27:37.460234 7517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:37.461197 7517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:37.462412 7517 hybrid_clock.cc:648] HybridClock initialized: now 1790450857462373 us; error 36 us; skew 500 ppm
I20260926 19:27:37.462733 7517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:37 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450857, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:37.470067 7517 init.cc:377] Logged in from keytab as oryx/127.1.39.2@KRBTEST.COM (short username oryx)
I20260926 19:27:37.471570 7517 webserver.cc:460] Webserver started at http://127.1.39.2:37133/ using document root <none> and password file <none>
I20260926 19:27:37.472285 7517 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:37.472424 7517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:37.472711 7517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:37.475652 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "f8f324248b154fbe954996cf05f74711"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "e3330a8003bad2e021c893ff90f18d66"
server_key_iv: "a79e97ed3aaca3bd50194114c11a9e23"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.476398 7517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "f8f324248b154fbe954996cf05f74711"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "e3330a8003bad2e021c893ff90f18d66"
server_key_iv: "a79e97ed3aaca3bd50194114c11a9e23"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.481422 7517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260926 19:27:37.484884 7532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.486285 7517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:27:37.486455 7517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "f8f324248b154fbe954996cf05f74711"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "e3330a8003bad2e021c893ff90f18d66"
server_key_iv: "a79e97ed3aaca3bd50194114c11a9e23"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.486663 7517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:37.499073 7517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:37.502948 7517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:37.503229 7517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:37.504108 7517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:37.505394 7517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:37.505487 7517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.505642 7517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:37.505713 7517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.544771 7517 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:33157
I20260926 19:27:37.544816 7644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:33157 every 8 connection(s)
I20260926 19:27:37.546419 7517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:37.551683 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7517
I20260926 19:27:37.551858 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:37.552428 1180 external_mini_cluster.cc:1428] Setting key c91920aa2990f8ca0be2b9d5badba74c
Sep 26 19:27:37 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450857, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:37.573510 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:37.549691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:34373 (local address 127.1.39.62:33725)
0926 19:27:37.549865 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:37.549869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:37.551247 (+ 1378us) server_negotiation.cc:407] Connection header received
0926 19:27:37.553775 (+ 2528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:37.553781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:37.553864 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:37.553979 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:37.555994 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.557074 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.559492 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.560244 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.560391 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:37.565438 (+ 5047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:37.565448 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:37.565455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:37.565498 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:37.569451 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:37.570404 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:37.570412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:37.570418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:37.570596 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:37.571096 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:37.571104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:37.571109 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:37.571489 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:37.571664 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:37.572754 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0926 19:27:37.573013 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:27:37.575119 7647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:37.550173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:34373)
0926 19:27:37.551080 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:37.551130 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:37.553515 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:37.554256 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:37.554268 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:37.555001 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:37.555793 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:37.555821 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.557217 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.557226 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:37.559343 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:37.559360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:37.560476 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:37.560486 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:37.560813 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:37.561784 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:37.561828 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:37.565134 (+ 3306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:37.569612 (+ 4478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:37.569624 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:37.569643 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:37.570283 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:37.570727 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:37.570735 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:37.570742 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:37.570977 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:37.571745 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:37.571755 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:37.572275 (+ 520us) client_negotiation.cc:769] Sending connection context
0926 19:27:37.572626 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 19:27:37.573236 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":380,"threads_started":1}
WARNING: no policy specified for oryx/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:27:37.576660 7645 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:37.576972 7645 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:37.577732 7645 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.3@KRBTEST.COM" created.
I20260926 19:27:37.579280 7322 ts_manager.cc:194] Registered new tserver with Master: f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:37.580485 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:34373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
I20260926 19:27:37.604665 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:37.925741 7650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:37.926007 7650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:37.926170 7650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:37.933216 7650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:37.933423 7650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:37.933698 7650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:37.943342 7650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:37.944622 7650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:37.945969 7650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:37.959355 7655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:37.959758 7656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:37.960263 7650 server_base.cc:1034] running on GCE node
W20260926 19:27:37.960386 7658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:37.961107 7650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:37.962146 7650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:37.963423 7650 hybrid_clock.cc:648] HybridClock initialized: now 1790450857963336 us; error 53 us; skew 500 ppm
I20260926 19:27:37.963829 7650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:37 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450857, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:37.967451 7650 init.cc:377] Logged in from keytab as oryx/127.1.39.3@KRBTEST.COM (short username oryx)
I20260926 19:27:37.968962 7650 webserver.cc:460] Webserver started at http://127.1.39.3:44423/ using document root <none> and password file <none>
I20260926 19:27:37.969736 7650 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:37.969877 7650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:37.970243 7650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:37.973057 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "8c1684fc580c4f68b0230019e1f0e6bb"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "08a383b691d5e71c728ba5c6c332eb5f"
server_key_iv: "acec58998a54713d444b434faa0c2043"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.973838 7650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "8c1684fc580c4f68b0230019e1f0e6bb"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "08a383b691d5e71c728ba5c6c332eb5f"
server_key_iv: "acec58998a54713d444b434faa0c2043"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.979619 7650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:27:37.983354 7665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:37.984772 7650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:37.984932 7650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "8c1684fc580c4f68b0230019e1f0e6bb"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "08a383b691d5e71c728ba5c6c332eb5f"
server_key_iv: "acec58998a54713d444b434faa0c2043"
server_key_version: "encryptionkey@0"
I20260926 19:27:37.985105 7650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:37.999997 7650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:38.004211 7650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:38.004490 7650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:38.005476 7650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:38.006788 7650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:38.006860 7650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:38.007011 7650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:38.007069 7650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:38.047946 7650 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:33309
I20260926 19:27:38.047967 7777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:33309 every 8 connection(s)
I20260926 19:27:38.049445 7650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:38.055963 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7650
I20260926 19:27:38.056156 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:38.056674 1180 external_mini_cluster.cc:1428] Setting key 2289a99cbbffcd3658a18fece918c175
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450857, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:38.073284 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.052628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:32987 (local address 127.1.39.62:33725)
0926 19:27:38.052807 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.052811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.054156 (+ 1345us) server_negotiation.cc:407] Connection header received
0926 19:27:38.056165 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.056174 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.056282 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.056414 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.058424 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.059952 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.061515 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.062397 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.062583 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.066842 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.066847 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.066850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.066880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.069648 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.070715 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.070721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.070727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.070927 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.071448 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.071452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.071456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.071798 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.071931 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.072871 (+ 940us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.073027 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:27:38.074988 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.053173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:32987)
0926 19:27:38.053998 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.054038 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.055786 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.056676 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.056688 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.057472 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.058240 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.058266 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.060107 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.060111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.061368 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.061381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.062611 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.062616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.062802 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.063504 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.063533 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.066619 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.069793 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.069807 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.069829 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.070573 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.071051 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.071055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.071061 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.071320 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.071925 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.071936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.072441 (+ 505us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.072766 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.073381 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":341,"threads_started":1}
I20260926 19:27:38.076402 7778 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:38.076438 7511 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
I20260926 19:27:38.076787 7778 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:38.077813 7778 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
I20260926 19:27:38.079366 7322 ts_manager.cc:194] Registered new tserver with Master: 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309)
I20260926 19:27:38.080137 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:32987
I20260926 19:27:38.084075 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.39.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.61.keytab.
Entry for principal oryx/127.1.39.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.61.keytab.
I20260926 19:27:38.134325 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:41101
--webserver_interface=127.1.39.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.61.keytab
--principal=oryx/127.1.39.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33725,127.1.39.61:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:38.458433 7783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:38.458740 7783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:38.458912 7783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:38.466249 7783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:38.466361 7783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:38.466436 7783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:38.466545 7783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:38.466625 7783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:38.476676 7783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33725,127.1.39.61:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.61.keytab
--principal=oryx/127.1.39.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.39.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
W20260926 19:27:38.477056 7783 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.39.62:33725,127.1.39.61:41101'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 19:27:38.478197 7783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:38.479533 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:38.498123 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:38.498302 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:38.499168 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:38.499643 7783 server_base.cc:1034] running on GCE node
I20260926 19:27:38.500705 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:38.502902 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:38.504192 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790450858504097 us; error 51 us; skew 500 ppm
I20260926 19:27:38.504626 7783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450858, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:38.509085 7783 init.cc:377] Logged in from keytab as oryx/127.1.39.61@KRBTEST.COM (short username oryx)
I20260926 19:27:38.510922 7783 webserver.cc:460] Webserver started at http://127.1.39.61:42341/ using document root <none> and password file <none>
I20260926 19:27:38.511651 7783 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:38.511780 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:38.512141 7783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:38.516311 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/instance:
uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
format_stamp: "Formatted at 2026-09-26 19:27:38 on dist-test-slave-9wb0"
server_key: "e15cd7a1c269494627a2ff777d6ab30b"
server_key_iv: "75f3865ec71379970c517e0c4495c5f4"
server_key_version: "encryptionkey@0"
I20260926 19:27:38.517457 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal/instance:
uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
format_stamp: "Formatted at 2026-09-26 19:27:38 on dist-test-slave-9wb0"
server_key: "e15cd7a1c269494627a2ff777d6ab30b"
server_key_iv: "75f3865ec71379970c517e0c4495c5f4"
server_key_version: "encryptionkey@0"
I20260926 19:27:38.522557 7783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:27:38.525955 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:38.527321 7783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:38.527496 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
format_stamp: "Formatted at 2026-09-26 19:27:38 on dist-test-slave-9wb0"
server_key: "e15cd7a1c269494627a2ff777d6ab30b"
server_key_iv: "75f3865ec71379970c517e0c4495c5f4"
server_key_version: "encryptionkey@0"
I20260926 19:27:38.527695 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:38.562268 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:38.566736 7783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:38.566948 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:38.583735 7645 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
I20260926 19:27:38.585803 7783 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.61:41101
I20260926 19:27:38.585853 7849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.61:41101 every 8 connection(s)
I20260926 19:27:38.587435 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
I20260926 19:27:38.589324 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7783
I20260926 19:27:38.592103 7850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:38.600244 7850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450858, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.61@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:38.612929 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.591652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.1.39.61:41101)
0926 19:27:38.592442 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.592454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.592493 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:27:38.594135 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.594156 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.594528 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.594843 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.596743 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.598299 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.600004 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.600845 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.600984 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.605698 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.605705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.605714 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.605747 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.608588 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.609462 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.609474 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.609480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.609645 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.610029 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.610034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.610037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.610707 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.611000 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.611453 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.611850 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":346,"threads_started":1}
I20260926 19:27:38.617581 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.603133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52270 (local address 127.1.39.62:33725)
0926 19:27:38.603730 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.603736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.604211 (+ 475us) server_negotiation.cc:407] Connection header received
0926 19:27:38.604414 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.604418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.604480 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.604604 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.605723 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.606445 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.607900 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.608301 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.608452 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.611842 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.611846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.611849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.611879 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.614847 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.615558 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.615565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.615568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.615702 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.616073 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.616078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.616081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.616375 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.616541 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.616970 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.617176 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":276,"threads_started":1}
I20260926 19:27:38.617623 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.603156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52270)
0926 19:27:38.604065 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.604122 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.604277 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.604670 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.604677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.605262 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.605571 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.605580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.606596 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.606603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.607723 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.607734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.608500 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.608506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.608679 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.609223 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.609238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.611660 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.614969 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.614979 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.614987 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.615432 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.615812 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.615818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.615822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.615956 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.616502 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.616510 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.616696 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.616882 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.617192 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":487,"threads_started":1}
I20260926 19:27:38.619495 7850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450858, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.61@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
I20260926 19:27:38.631419 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.620157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39124)
0926 19:27:38.620353 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.620375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.620472 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.620781 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.620785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.621129 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.621252 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.621260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.622241 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.622248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.623098 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.623108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.623762 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.623767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.623936 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.624449 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.624465 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.626460 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.629007 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.629024 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.629028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.629445 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.629820 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.629827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.629832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.629957 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.630433 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.630437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.630588 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.630717 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.631002 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":4}
I20260926 19:27:38.631464 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.620311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.1.39.61:41101)
0926 19:27:38.620451 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.620457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.620474 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:27:38.620608 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.620613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.620672 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.620767 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.621409 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.622108 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.623229 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.623622 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.623750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.626622 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.626628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.626639 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.626665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.628883 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.629553 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.629560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.629564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.629708 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.630069 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.630075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.630083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.630323 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.630468 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.630801 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.631137 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":48}
I20260926 19:27:38.636438 7850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Bootstrap starting.
I20260926 19:27:38.639560 7850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:38.640496 7850 log.cc:824] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:38.643026 7850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: No bootstrap required, opened a new log
I20260926 19:27:38.649888 7850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } }
I20260926 19:27:38.650195 7850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:38.650270 7850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49931b8ce6234c9fbaf8dcd18c3aeb99, State: Initialized, Role: FOLLOWER
I20260926 19:27:38.650833 7850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } }
I20260926 19:27:38.651618 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } } }
I20260926 19:27:38.651933 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:38.652580 7850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:27:38.661178 7868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:27:38.661358 7868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:27:38.663642 7783 master_runner.cc:428] Detected that this master 49931b8ce6234c9fbaf8dcd18c3aeb99 is joining an existing cluster
I20260926 19:27:38.663749 7783 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.39.61:41101
I20260926 19:27:38.692066 7857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:38.692399 7857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } }
WARNING: no policy specified for oryx/127.1.39.60@KRBTEST.COM; defaulting to no policy
I20260926 19:27:38.693974 7857 leader_election.cc:288] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725)
I20260926 19:27:38.694850 7332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0b3c26acf0547b9a575729235755a20" is_pre_election: true
I20260926 19:27:38.695253 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.668398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.1.39.61:41101)
0926 19:27:38.668572 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.668578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.678748 (+ 10170us) server_negotiation.cc:407] Connection header received
0926 19:27:38.678904 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.678912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.678991 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.679109 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.680406 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.681430 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.683503 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.684154 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.684243 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.687468 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.687481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.687487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.687526 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.691615 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.692573 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.692579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.692582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.692728 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.693245 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.693249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.693251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.693605 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.693743 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.694492 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.694839 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":50}
I20260926 19:27:38.695739 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.668003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39128)
0926 19:27:38.678634 (+ 10631us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.678652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.678783 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.679282 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.679288 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.679949 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.680271 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.680281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.681634 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.681639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.683352 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.683368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.684553 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.684561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.684749 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.685824 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.685849 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.687197 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.691808 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.691812 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.691816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.692452 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.692981 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.692985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.692988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.693134 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.693876 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.693880 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.694053 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.694367 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.694665 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10513,"mutex_wait_us":41,"thread_start_us":9507,"threads_started":1}
I20260926 19:27:38.696142 7801 leader_election.cc:302] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99; no voters: d0b3c26acf0547b9a575729235755a20
I20260926 19:27:38.696573 7857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.60@KRBTEST.COM" created.
I20260926 19:27:38.700074 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.668428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.1.39.62:33725)
0926 19:27:38.668594 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.668602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.677960 (+ 9358us) server_negotiation.cc:407] Connection header received
0926 19:27:38.678294 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.678305 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.678411 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.678568 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.680260 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.681757 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.683753 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.684389 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.684494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.687872 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.687878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.687883 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.687933 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.691610 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.697445 (+ 5835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.697450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.697453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.697628 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.698124 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.698129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.698133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.698551 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.698691 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.699330 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.699608 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:27:38.701354 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.667970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52274)
0926 19:27:38.677809 (+ 9839us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.677837 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.678097 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.678777 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.678782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.679637 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.680053 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.680065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.681957 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.681962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.683621 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.683632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.684756 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.684761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.684998 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.686596 (+ 1598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.686617 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.687656 (+ 1039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.696778 (+ 9122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.696785 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.696790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.697288 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.697800 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.697806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.697810 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.697987 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.698778 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.698785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.699017 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.700295 (+ 1278us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.700735 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9672,"mutex_wait_us":74,"thread_start_us":10353,"threads_started":1}
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.62@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
I20260926 19:27:38.721544 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.707820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39134 (local address 127.1.39.61:41101)
0926 19:27:38.708002 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.708008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.708248 (+ 240us) server_negotiation.cc:407] Connection header received
0926 19:27:38.708570 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.708579 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.708701 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.708760 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.710044 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.710803 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.712094 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.712554 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.712699 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.715852 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.715857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.715861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.715892 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.718777 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.719512 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.719519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.719525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.719665 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.720036 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.720043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.720047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.720339 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.720485 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.720907 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.721138 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:27:38.721596 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.707557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39134)
0926 19:27:38.708134 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.708173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.708367 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.708977 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.708983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.709645 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.709889 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.709903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.710947 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.710953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.711963 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.711973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.712713 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.712719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.712940 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.713600 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.713617 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.715668 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.718969 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.718981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.718989 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.719378 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.719773 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.719778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.719782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.719932 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.720470 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.720478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.720642 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.720825 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.721156 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":447,"threads_started":1}
I20260926 19:27:38.723691 7322 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.39.61:41101: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
member_type: NON_VOTER
last_known_addr {
host: "127.1.39.61"
port: 41101
}
attrs {
promote: true
}
}
dest_uuid: "d0b3c26acf0547b9a575729235755a20"
cas_config_opid_index: -1
I20260926 19:27:38.724735 7322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.60.keytab.
Entry for principal oryx/127.1.39.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.60.keytab.
I20260926 19:27:38.728758 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:38.727927 7881 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } }
I20260926 19:27:38.729677 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:40345
--webserver_interface=127.1.39.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.60.keytab
--principal=oryx/127.1.39.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:27:38.732542 7825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 19:27:38.737627 7881 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.39.61:41101
I20260926 19:27:38.737788 7883 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:38.738258 7881 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } }
I20260926 19:27:38.738468 7881 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:38.747493 7783 master.cc:537] Master@127.1.39.61:41101 shutting down...
W20260926 19:27:38.748610 7307 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 19:27:38.751842 7307 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 -> Peer 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61:41101): Couldn't send request to peer 49931b8ce6234c9fbaf8dcd18c3aeb99. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 19:27:38.759886 7783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:38.760304 7783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:38.760488 7783 tablet_replica.cc:330] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: stopping tablet replica
I20260926 19:27:38.766578 7783 master.cc:552] Master@127.1.39.61:41101 shutdown complete.
I20260926 19:27:38.766719 7783 master_runner.cc:305] Clearing existing system tablet
I20260926 19:27:38.767100 7783 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:38.767210 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:38.769467 7783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:27:38.770974 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:38.771287 7783 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:38.771577 7783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 19:27:38.771668 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
format_stamp: "Formatted at 2026-09-26 19:27:38 on dist-test-slave-9wb0"
server_key: "e15cd7a1c269494627a2ff777d6ab30b"
server_key_iv: "75f3865ec71379970c517e0c4495c5f4"
server_key_version: "encryptionkey@0"
I20260926 19:27:38.775578 7783 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:38.780190 7783 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 19:27:38.780369 7783 log.cc:1196] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 19:27:38.780761 7783 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Deleting consensus metadata
I20260926 19:27:38.781013 7783 master_runner.cc:315] Copying system tablet from 127.1.39.62:33725
I20260926 19:27:38.783386 7783 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: tablet copy: Beginning tablet copy session from remote peer at address 127.1.39.62:33725
I20260926 19:27:38.796455 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.784131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.1.39.62:33725)
0926 19:27:38.784345 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:38.784354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:38.784844 (+ 490us) server_negotiation.cc:407] Connection header received
0926 19:27:38.785054 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:38.785059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:38.785160 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:38.785270 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:38.786126 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.787065 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.788529 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.789083 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.789220 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.791186 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:38.791192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:38.791203 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:38.791232 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:38.793889 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.794503 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.794510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.794514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.794633 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:38.795009 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:38.795014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:38.795018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:38.795276 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:38.795425 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:38.795851 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 19:27:38.796076 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:27:38.796548 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:38.784104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52280)
0926 19:27:38.784733 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:38.784752 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:38.784916 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:38.785358 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:38.785363 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:38.785773 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:38.785983 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.785992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.787199 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.787205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.788389 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:38.788400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:38.789226 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:38.789232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:38.789466 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:38.790166 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:38.790184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:38.790927 (+ 743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:38.794026 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.794033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.794037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.794374 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.794739 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:38.794744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:38.794748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:38.794894 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:38.795390 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:38.795395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:38.795564 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 19:27:38.795743 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 19:27:38.796078 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":386,"threads_started":1}
I20260926 19:27:38.797688 7342 tablet_copy_service.cc:140] P d0b3c26acf0547b9a575729235755a20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 49931b8ce6234c9fbaf8dcd18c3aeb99 ({username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:52280)
I20260926 19:27:38.797909 7342 tablet_copy_service.cc:161] P d0b3c26acf0547b9a575729235755a20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 49931b8ce6234c9fbaf8dcd18c3aeb99 at {username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:52280: session id = 49931b8ce6234c9fbaf8dcd18c3aeb99-00000000000000000000000000000000
I20260926 19:27:38.801205 7342 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 19:27:38.803093 7783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 19:27:38.807547 7783 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: tablet copy: Starting download of 0 data blocks...
I20260926 19:27:38.807850 7783 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: tablet copy: Starting download of 1 WAL segments...
I20260926 19:27:38.810156 7783 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 19:27:38.813341 7342 tablet_copy_service.cc:342] P d0b3c26acf0547b9a575729235755a20: Request end of tablet copy session 49931b8ce6234c9fbaf8dcd18c3aeb99-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:52280
I20260926 19:27:38.813544 7342 tablet_copy_service.cc:433] P d0b3c26acf0547b9a575729235755a20: ending tablet copy session 49931b8ce6234c9fbaf8dcd18c3aeb99-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 49931b8ce6234c9fbaf8dcd18c3aeb99
I20260926 19:27:38.823375 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:38.826071 7895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:38.830891 7896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:38.843135 7898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:38.843667 7783 server_base.cc:1034] running on GCE node
I20260926 19:27:38.844074 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:38.845583 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:38.846825 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790450858846756 us; error 36 us; skew 500 ppm
I20260926 19:27:38.847054 7783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:38 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450858, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:38.849270 7783 init.cc:377] Logged in from keytab as oryx/127.1.39.61@KRBTEST.COM (short username oryx)
I20260926 19:27:38.850484 7783 webserver.cc:460] Webserver started at http://127.1.39.61:33473/ using document root <none> and password file <none>
I20260926 19:27:38.850826 7783 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:38.850906 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:38.854005 7783 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 19:27:38.855929 7905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:38.856393 7783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260926 19:27:38.856518 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
format_stamp: "Formatted at 2026-09-26 19:27:38 on dist-test-slave-9wb0"
server_key: "e15cd7a1c269494627a2ff777d6ab30b"
server_key_iv: "75f3865ec71379970c517e0c4495c5f4"
server_key_version: "encryptionkey@0"
I20260926 19:27:38.856678 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:38.879441 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:38.883852 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:38.907513 7783 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.61:41101
I20260926 19:27:38.907665 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.61:41101 every 8 connection(s)
I20260926 19:27:38.908489 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
I20260926 19:27:38.913722 7957 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:27:38.917578 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Bootstrap starting.
I20260926 19:27:38.934376 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:27:38.934873 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Bootstrap complete.
I20260926 19:27:38.935909 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } }
I20260926 19:27:38.936247 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49931b8ce6234c9fbaf8dcd18c3aeb99, State: Initialized, Role: LEARNER
I20260926 19:27:38.936539 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } }
I20260926 19:27:38.937407 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } }
I20260926 19:27:38.937772 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: LEARNER
I20260926 19:27:38.937647 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:38.957492 7971 catalog_manager.cc:1259] Loaded cluster ID: 0293c0558520431e87dd0a0ffc992310
I20260926 19:27:38.957603 7971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: loading cluster ID for follower catalog manager: success
I20260926 19:27:38.961685 7971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: acquiring CA information for follower catalog manager: success
I20260926 19:27:38.963965 7971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:27:39.083482 7778 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
W20260926 19:27:39.123847 7884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:39.124173 7884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:39.124329 7884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:39.131657 7884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:39.131825 7884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:39.131930 7884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:39.132025 7884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:39.132104 7884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:39.142860 7884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.60.keytab
--principal=oryx/127.1.39.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.39.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:39.144279 7884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:39.145720 7884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:39.158349 7976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:39.158437 7977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:39.158365 7979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:39.159691 7884 server_base.cc:1034] running on GCE node
I20260926 19:27:39.160516 7884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:39.161943 7884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:39.163188 7884 hybrid_clock.cc:648] HybridClock initialized: now 1790450859163134 us; error 63 us; skew 500 ppm
I20260926 19:27:39.163547 7884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:39.167089 7884 init.cc:377] Logged in from keytab as oryx/127.1.39.60@KRBTEST.COM (short username oryx)
I20260926 19:27:39.168602 7884 webserver.cc:460] Webserver started at http://127.1.39.60:35307/ using document root <none> and password file <none>
I20260926 19:27:39.169334 7884 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:39.169485 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:39.169795 7884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:39.172776 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/instance:
uuid: "c2064901f9184f989adaca33d7a46085"
format_stamp: "Formatted at 2026-09-26 19:27:39 on dist-test-slave-9wb0"
server_key: "1b35b0227166b4bdbf662bd3f28f2833"
server_key_iv: "0989c053d7f77171fec1ba2c6bd145bc"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.173648 7884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal/instance:
uuid: "c2064901f9184f989adaca33d7a46085"
format_stamp: "Formatted at 2026-09-26 19:27:39 on dist-test-slave-9wb0"
server_key: "1b35b0227166b4bdbf662bd3f28f2833"
server_key_iv: "0989c053d7f77171fec1ba2c6bd145bc"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.178617 7884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:27:39.182031 7986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:39.183506 7884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:39.183706 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "c2064901f9184f989adaca33d7a46085"
format_stamp: "Formatted at 2026-09-26 19:27:39 on dist-test-slave-9wb0"
server_key: "1b35b0227166b4bdbf662bd3f28f2833"
server_key_iv: "0989c053d7f77171fec1ba2c6bd145bc"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.183920 7884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:39.203138 7884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:39.207280 7884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:39.207556 7884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:39.227135 7884 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.60:40345
I20260926 19:27:39.227162 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.60:40345 every 8 connection(s)
I20260926 19:27:39.228507 7884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
I20260926 19:27:39.230233 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 7884
I20260926 19:27:39.233040 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:39.240497 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.60@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:39.254276 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.232334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.1.39.60:40345)
0926 19:27:39.233127 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.233142 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.233191 (+ 49us) server_negotiation.cc:407] Connection header received
0926 19:27:39.234829 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.234850 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.235247 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.235640 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.237145 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.239121 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.240442 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.241103 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.241294 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.246358 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.246365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.246373 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.246423 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.250060 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.250695 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.250705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.250716 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.250930 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.251410 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.251414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.251417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.252008 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.252220 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.253038 (+ 818us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.253365 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":340,"threads_started":1}
I20260926 19:27:39.258096 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.242915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52284 (local address 127.1.39.62:33725)
0926 19:27:39.243149 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.243157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.243769 (+ 612us) server_negotiation.cc:407] Connection header received
0926 19:27:39.243934 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.243937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.244014 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.244099 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.245488 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.246498 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.248542 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.248904 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.248993 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.252532 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.252537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.252540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.252569 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.255256 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.255900 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.255905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.255908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.256051 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.256389 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.256394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.256397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.256779 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.256947 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.257512 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.257684 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 19:27:39.258354 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.242845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52284)
0926 19:27:39.243608 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.243662 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.243828 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.244193 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.244201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.245006 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.245236 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.245244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.246650 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.246656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.248409 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.248423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.249117 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.249124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.249331 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.249868 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.249917 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.252360 (+ 2443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.255394 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.255400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.255403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.255786 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.256156 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.256159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.256161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.256286 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.257005 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.257010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.257216 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.257421 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.257638 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":356,"threads_started":1}
I20260926 19:27:39.259466 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.60@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
I20260926 19:27:39.271836 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.260072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39144)
0926 19:27:39.260254 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.260273 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.260389 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.260919 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.260923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.261277 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.261469 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.261477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.262686 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.262694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.263660 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.263671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.264365 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.264371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.264540 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.265081 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.265097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.266969 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.269597 (+ 2628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.269603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.269607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.269956 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.270335 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.270340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.270344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.270470 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.270939 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.270943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.271083 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.271225 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.271485 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:27:39.271941 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.260218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.1.39.61:41101)
0926 19:27:39.260673 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.260677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.260690 (+ 13us) server_negotiation.cc:407] Connection header received
0926 19:27:39.260734 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.260737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.260792 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.260904 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.261603 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.262532 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.263809 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.264213 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.264345 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.267155 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.267160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.267163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.267191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.269479 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.270071 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.270078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.270081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.270227 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.270574 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.270581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.270584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.270825 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.270981 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.271310 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.271508 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":231,"threads_started":1}
I20260926 19:27:39.272856 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.60@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
I20260926 19:27:39.285836 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.273479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.0.0.1:48808)
0926 19:27:39.273711 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.273725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.273829 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.274084 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.274087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.274413 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.274525 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.274532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.275733 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.275741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.277506 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.277521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.278574 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.278581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.278809 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.279283 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.279298 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.281110 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.283914 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.283918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.283920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.284242 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.284583 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.284586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.284588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.284716 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.285124 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.285126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.285230 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.285393 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.285577 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:27:39.286016 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.273717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.1.39.60:40345)
0926 19:27:39.273856 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.273863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.273882 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:39.273922 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.273927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.273978 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.274071 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.274659 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.275620 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.277655 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.278397 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.278523 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.281255 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.281259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.281262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.281288 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.283806 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.284360 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.284371 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.284373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.284482 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.284802 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.284805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.284807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.285034 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.285146 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.285443 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.285689 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":51}
I20260926 19:27:39.290894 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Bootstrap starting.
I20260926 19:27:39.293998 8038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:39.294961 8038 log.cc:824] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:39.297988 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: No bootstrap required, opened a new log
I20260926 19:27:39.304931 8038 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } }
I20260926 19:27:39.305246 8038 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:39.305398 8038 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2064901f9184f989adaca33d7a46085, State: Initialized, Role: FOLLOWER
I20260926 19:27:39.306005 8038 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } }
I20260926 19:27:39.306792 8045 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } } }
I20260926 19:27:39.307127 8045 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:39.307891 8038 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:39.318943 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.311540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.1.39.61:41101)
0926 19:27:39.311755 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.311762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.312095 (+ 333us) server_negotiation.cc:407] Connection header received
0926 19:27:39.312329 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.312337 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.312453 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.312601 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:39.313080 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.314024 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.316573 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.317705 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.317838 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.318056 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.318557 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.318695 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:27:39.318997 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.311225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39152)
0926 19:27:39.311972 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.311995 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.312144 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.312658 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.312666 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.312702 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:39.312931 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.312943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.314178 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.314183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.316391 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.316406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.318160 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.318166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.318348 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.318411 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.318517 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.318667 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":454,"threads_started":1}
W20260926 19:27:39.319548 8058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:27:39.319679 8058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:27:39.320202 7931 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 19:27:39.320930 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b3c26acf0547b9a575729235755a20. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } }
I20260926 19:27:39.321162 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: LEARNER
I20260926 19:27:39.322082 7884 master_runner.cc:428] Detected that this master c2064901f9184f989adaca33d7a46085 is joining an existing cluster
I20260926 19:27:39.322175 7884 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.39.60:40345
I20260926 19:27:39.344714 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.326019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.1.39.62:33725)
0926 19:27:39.326203 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.326209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.327398 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 19:27:39.327658 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.327665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.327739 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.327807 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.329388 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.330615 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.333329 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.333936 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.334033 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.337385 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.337391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.337397 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.337435 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.341265 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.342116 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.342122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.342126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.342388 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.342883 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.342890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.342894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.343240 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.343360 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.344070 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.344294 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:27:39.345170 8043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.325977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39168 (local address 127.1.39.61:41101)
0926 19:27:39.326214 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.326219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.327752 (+ 1533us) server_negotiation.cc:407] Connection header received
0926 19:27:39.327978 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.327983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.328054 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.328120 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.330107 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.331475 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.333931 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.334992 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.335086 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.337763 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.337769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.337774 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.337811 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.341973 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.342777 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.342784 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.342789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.342979 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.343472 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.343477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.343480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.343823 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.343968 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.344605 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.344828 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 19:27:39.346403 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.327028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:39168)
0926 19:27:39.327670 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.327687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.327834 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.328354 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.328361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.329349 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.329891 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.329908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.331725 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.331733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.333737 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.333751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.335389 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.335396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.335580 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.336205 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.336235 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.337579 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.342119 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.342138 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.342141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.342628 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.343136 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.343144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.343149 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.343343 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.344056 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.344062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.344307 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.345428 (+ 1121us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.345783 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":29,"thread_start_us":463,"threads_started":1}
I20260926 19:27:39.346444 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.326194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52292)
0926 19:27:39.327298 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.327318 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.327506 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.328013 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.328018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.328829 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.329190 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.329202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.330794 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.330799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.333165 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.333175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.334769 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.334776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.334968 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.335681 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.335709 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.337143 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.341459 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.341464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.341480 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.341963 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.342550 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.342555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.342559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.342759 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.343503 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.343510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.343734 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.345477 (+ 1743us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.345970 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"mutex_wait_us":818,"thread_start_us":1056,"threads_started":1}
I20260926 19:27:39.349669 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.328112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.1.39.60:40345)
0926 19:27:39.328299 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.328309 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.332969 (+ 4660us) server_negotiation.cc:407] Connection header received
0926 19:27:39.333073 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.333077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.333144 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.333241 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.335041 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.336163 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.338212 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.338924 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.339015 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.341902 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.341909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.341916 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.342022 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.345783 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.346744 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.346752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.346754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.347534 (+ 780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.347922 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.347926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.347929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.348298 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.348439 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.348934 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.349280 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":59,"spinlock_wait_cycles":27520}
I20260926 19:27:39.350324 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.327834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.0.0.1:48816)
0926 19:27:39.332863 (+ 5029us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.332880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.332988 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.333441 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.333446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.334123 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.334892 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.334904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.336383 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.336389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.338081 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.338093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.339270 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.339278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.339479 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.340070 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.340092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.341694 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.345932 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.345938 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.345946 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.346484 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.347687 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.347690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.347694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.347820 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.348502 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.348509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.348673 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.348824 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.349129 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4885,"thread_start_us":4745,"threads_started":1}
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450856, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.62@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
I20260926 19:27:39.360764 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7383
I20260926 19:27:39.366525 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.352579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.0.0.1:48832)
0926 19:27:39.352781 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.352800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.352897 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.353332 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.353339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.353785 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.354197 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.354207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.355207 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.355213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.356223 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.356235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.357021 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.357026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.357204 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.357879 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.357896 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.360285 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.364041 (+ 3756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.364046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.364050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.364414 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.364772 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.364778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.364781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.364912 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.365543 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.365548 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.365720 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.365882 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.366160 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 19:27:39.366552 8041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.352837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48832 (local address 127.1.39.60:40345)
0926 19:27:39.353003 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.353008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.353025 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:27:39.353100 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.353104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.353178 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.353288 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.354344 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.355082 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.356366 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.356857 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.357008 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.360624 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.360631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.360636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.360676 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.363902 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.364531 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.364538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.364542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.364663 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.365016 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.365022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.365026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.365395 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.365563 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.365995 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.366226 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:27:39.368358 7322 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.39.60:40345: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c2064901f9184f989adaca33d7a46085"
member_type: NON_VOTER
last_known_addr {
host: "127.1.39.60"
port: 40345
}
attrs {
promote: true
}
}
dest_uuid: "d0b3c26acf0547b9a575729235755a20"
cas_config_opid_index: 5
I20260926 19:27:39.369412 7322 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } }
I20260926 19:27:39.371227 8047 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c2064901f9184f989adaca33d7a46085 (127.1.39.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.1@KRBTEST.COM
I20260926 19:27:39.376454 7931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 19:27:39.376700 8013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:39.377219 8047 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.39.60:40345
I20260926 19:27:39.380507 8068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 19:27:39.384841 8069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.385223 8069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:39.387549 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.361752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.1.39.1:33623)
0926 19:27:39.362415 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.362422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.362446 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:27:39.362532 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.362543 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.362827 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.363019 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.365272 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.367244 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.368742 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.369411 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.369543 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.377874 (+ 8331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.377889 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.377907 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.377956 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.381979 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.382840 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.382848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.382852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.383025 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.383610 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.383618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.383623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.384167 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.386488 (+ 2321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.386645 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.386967 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":308,"threads_started":1}
I20260926 19:27:39.389544 7506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:39.393741 7931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c2064901f9184f989adaca33d7a46085 (127.1.39.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.397653 7884 master.cc:537] Master@127.1.39.60:40345 shutting down...
I20260926 19:27:39.397755 8013 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 19:27:39.398715 8069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:39.399669 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: NON_VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: true } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.400012 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: LEARNER
I20260926 19:27:39.402423 8047 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: attempting to promote NON_VOTER 49931b8ce6234c9fbaf8dcd18c3aeb99 to VOTER
I20260926 19:27:39.403231 8047 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } }
I20260926 19:27:39.406421 7931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 LEARNER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 19:27:39.406930 8068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 19:27:39.410977 7884 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:39.410483 8069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.410933 7931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.411404 7884 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:39.411626 7884 tablet_replica.cc:330] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: stopping tablet replica
I20260926 19:27:39.414927 8047 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.415205 8047 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:39.415545 8069 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.415761 8069 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
W20260926 19:27:39.416754 7307 consensus_peers.cc:487] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 -> Peer c2064901f9184f989adaca33d7a46085 (127.1.39.60:40345): Couldn't send request to peer c2064901f9184f989adaca33d7a46085. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 19:27:39.416687 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.416951 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:39.421183 7884 master.cc:552] Master@127.1.39.60:40345 shutdown complete.
I20260926 19:27:39.421343 7884 master_runner.cc:305] Clearing existing system tablet
I20260926 19:27:39.421824 7884 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:39.421977 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:39.424911 7884 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 19:27:39.426812 8076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:39.427186 7884 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:39.427439 7884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 19:27:39.427529 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "c2064901f9184f989adaca33d7a46085"
format_stamp: "Formatted at 2026-09-26 19:27:39 on dist-test-slave-9wb0"
server_key: "1b35b0227166b4bdbf662bd3f28f2833"
server_key_iv: "0989c053d7f77171fec1ba2c6bd145bc"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.432447 7884 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:39.439026 7884 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 19:27:39.439181 7884 log.cc:1196] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 19:27:39.439601 7884 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Deleting consensus metadata
I20260926 19:27:39.439833 7884 master_runner.cc:315] Copying system tablet from 127.1.39.62:33725
I20260926 19:27:39.442417 7884 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: tablet copy: Beginning tablet copy session from remote peer at address 127.1.39.62:33725
I20260926 19:27:39.458848 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.443587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.1.39.62:33725)
0926 19:27:39.443848 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.443855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.444096 (+ 241us) server_negotiation.cc:407] Connection header received
0926 19:27:39.444341 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.444347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.444419 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.444480 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:39.445613 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.446732 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.448556 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.449228 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.449334 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.451698 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:39.451705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:39.451710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:39.451745 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:39.455330 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.456143 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.456148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.456153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.456287 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:39.456759 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:39.456764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:39.456768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:39.457096 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:39.457212 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.458267 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.458497 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 19:27:39.459772 8082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.443541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:52298)
0926 19:27:39.443995 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.444017 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.444207 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.444670 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.444675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.445204 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:39.445451 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.445462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.446926 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.446932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.448407 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.448419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.449552 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.449558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.449719 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.450457 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:39.450483 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:39.451467 (+ 984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:39.455498 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.455503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.455508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.456002 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.456470 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:39.456476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:39.456480 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:39.456638 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:39.457334 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:39.457339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:39.457553 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.459028 (+ 1475us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.459348 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":7825,"threads_started":1}
I20260926 19:27:39.460484 7342 tablet_copy_service.cc:140] P d0b3c26acf0547b9a575729235755a20: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c2064901f9184f989adaca33d7a46085 ({username='oryx', principal='oryx/127.1.39.60@KRBTEST.COM'} at 127.0.0.1:52298)
I20260926 19:27:39.460638 7342 tablet_copy_service.cc:161] P d0b3c26acf0547b9a575729235755a20: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c2064901f9184f989adaca33d7a46085 at {username='oryx', principal='oryx/127.1.39.60@KRBTEST.COM'} at 127.0.0.1:52298: session id = c2064901f9184f989adaca33d7a46085-00000000000000000000000000000000
I20260926 19:27:39.464440 7342 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 19:27:39.466161 7884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 19:27:39.472901 7884 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: tablet copy: Starting download of 0 data blocks...
I20260926 19:27:39.473181 7884 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: tablet copy: Starting download of 1 WAL segments...
I20260926 19:27:39.475545 7884 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 19:27:39.479804 7342 tablet_copy_service.cc:342] P d0b3c26acf0547b9a575729235755a20: Request end of tablet copy session c2064901f9184f989adaca33d7a46085-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.39.60@KRBTEST.COM'} at 127.0.0.1:52298
I20260926 19:27:39.479943 7342 tablet_copy_service.cc:433] P d0b3c26acf0547b9a575729235755a20: ending tablet copy session c2064901f9184f989adaca33d7a46085-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c2064901f9184f989adaca33d7a46085
I20260926 19:27:39.490463 7884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:39.493144 8083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:39.501874 8084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:39.509471 7884 server_base.cc:1034] running on GCE node
W20260926 19:27:39.509788 8086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:39.510237 7884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:39.510882 7884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:39.512090 7884 hybrid_clock.cc:648] HybridClock initialized: now 1790450859512037 us; error 32 us; skew 500 ppm
I20260926 19:27:39.512323 7884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:39.514443 7884 init.cc:377] Logged in from keytab as oryx/127.1.39.60@KRBTEST.COM (short username oryx)
I20260926 19:27:39.515690 7884 webserver.cc:460] Webserver started at http://127.1.39.60:44901/ using document root <none> and password file <none>
I20260926 19:27:39.515997 7884 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:39.516096 7884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:39.519136 7884 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 19:27:39.520985 8093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:39.521458 7884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260926 19:27:39.521575 7884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "c2064901f9184f989adaca33d7a46085"
format_stamp: "Formatted at 2026-09-26 19:27:39 on dist-test-slave-9wb0"
server_key: "1b35b0227166b4bdbf662bd3f28f2833"
server_key_iv: "0989c053d7f77171fec1ba2c6bd145bc"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.521752 7884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:39.532804 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7383
I20260926 19:27:39.541631 7884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:39.545688 7884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:39.556869 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:33623
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:33725
--webserver_port=42053
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:27:39.571704 7884 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.60:40345
I20260926 19:27:39.571761 8145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.60:40345 every 8 connection(s)
I20260926 19:27:39.572690 7884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
I20260926 19:27:39.578889 8146 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:27:39.583351 8146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Bootstrap starting.
I20260926 19:27:39.601284 8146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:27:39.601794 8146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: Bootstrap complete.
I20260926 19:27:39.602788 8146 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } }
I20260926 19:27:39.603055 8146 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c2064901f9184f989adaca33d7a46085, State: Initialized, Role: LEARNER
I20260926 19:27:39.603322 8146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } }
I20260926 19:27:39.604287 8146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:39.604039 8149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.604570 8149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: LEARNER
I20260926 19:27:39.621423 8160 catalog_manager.cc:1259] Loaded cluster ID: 0293c0558520431e87dd0a0ffc992310
I20260926 19:27:39.621519 8160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: loading cluster ID for follower catalog manager: success
I20260926 19:27:39.624532 8160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: acquiring CA information for follower catalog manager: success
I20260926 19:27:39.626737 8160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:27:39.805912 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.798822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.1.39.60:40345)
0926 19:27:39.799494 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:39.799498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:39.799519 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:39.799578 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:39.799581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:39.799639 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:39.799734 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:39.800269 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.801468 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.804072 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.804886 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.805041 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.805222 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:39.805573 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 19:27:39.805667 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":433,"threads_started":1}
I20260926 19:27:39.806169 8053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:39.798637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.0.0.1:48840)
0926 19:27:39.798864 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:39.798890 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:39.799057 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:39.799807 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:39.799816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:39.799856 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:39.800114 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.800130 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.801617 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.801626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.803901 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:39.803918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:39.805056 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:39.805066 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:39.805352 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:39.805442 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 19:27:39.805552 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 19:27:39.805753 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:27:39.807087 8120 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 19:27:39.807785 8149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0b3c26acf0547b9a575729235755a20. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:39.808095 8149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: LEARNER
W20260926 19:27:39.915777 8124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:39.916088 8124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:39.916237 8124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:39.925046 8124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:39.925197 8124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:39.925423 8124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:39.935001 8124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:33623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=42053
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:39.936300 8124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:39.937773 8124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:39.951339 8167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:39.951399 8169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:39.951424 8166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:39.952365 8124 server_base.cc:1034] running on GCE node
I20260926 19:27:39.952946 8124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:39.953922 8124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:39.955155 8124 hybrid_clock.cc:648] HybridClock initialized: now 1790450859955110 us; error 59 us; skew 500 ppm
I20260926 19:27:39.955473 8124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:39 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:39.958889 8124 init.cc:377] Logged in from keytab as oryx/127.1.39.1@KRBTEST.COM (short username oryx)
I20260926 19:27:39.960449 8124 webserver.cc:460] Webserver started at http://127.1.39.1:42053/ using document root <none> and password file <none>
I20260926 19:27:39.961151 8124 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:39.961350 8124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:39.966696 8124 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 19:27:39.969264 8176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:39.970700 8124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:27:39.970866 8124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "827c49b220cf4c4a98ae0a3c38f06420"
format_stamp: "Formatted at 2026-09-26 19:27:36 on dist-test-slave-9wb0"
server_key: "44e0a1c0ff6149484558c265e05f8dab"
server_key_iv: "55cf394afa6a2ebea6803297001868c1"
server_key_version: "encryptionkey@0"
I20260926 19:27:39.972183 8124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:39.990833 8124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:39.994786 8124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:39.995086 8124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:39.996080 8124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:39.997480 8124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:39.997597 8124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:39.997723 8124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:39.997816 8124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:40.035079 8124 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33623
I20260926 19:27:40.035169 8288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33623 every 8 connection(s)
I20260926 19:27:40.036556 8124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:40.041779 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8124
I20260926 19:27:40.042120 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7517
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450859, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.2@KRBTEST.COM
I20260926 19:27:40.078140 8293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.039598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:50827 (local address 127.1.39.62:33725)
0926 19:27:40.040458 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.040462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.048828 (+ 8366us) server_negotiation.cc:407] Connection header received
0926 19:27:40.050925 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.050934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.051050 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.051144 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.060097 (+ 8953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.061240 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.063358 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.063967 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.064091 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.069381 (+ 5290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.069390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.069397 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.069441 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.073983 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075194 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.075199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.075203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.075335 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075915 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.075919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.075922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.076343 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.076473 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.077555 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.077756 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":649,"threads_started":1}
I20260926 19:27:40.078205 8161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.039613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:53905 (local address 127.1.39.60:40345)
0926 19:27:40.039800 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.039806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.041257 (+ 1451us) server_negotiation.cc:407] Connection header received
0926 19:27:40.056435 (+ 15178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.056442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.056528 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.056619 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.057794 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.059221 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.062664 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.063283 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.063371 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.069865 (+ 6494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.069872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.069876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.069916 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.073917 (+ 4001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075170 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.075176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.075180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.075332 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075928 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.075934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.075937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.076304 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.076435 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.077573 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.077835 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:27:40.079295 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.041750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.1.39.1:57415)
0926 19:27:40.058762 (+ 17012us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.058776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.058921 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.059279 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.059282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.059889 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.060240 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.060252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.061650 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.061655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.063319 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.063332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.064458 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.064463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.064697 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.065349 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.065399 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.070823 (+ 5424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.074902 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.074908 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.074911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075379 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.075792 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.075799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075969 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.076585 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.076588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.077107 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.077442 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.077918 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16920,"mutex_wait_us":148,"spinlock_wait_cycles":35584,"thread_start_us":16409,"threads_started":1}
I20260926 19:27:40.079324 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.040091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.1.39.1:53905)
0926 19:27:40.041082 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.041141 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.050098 (+ 8957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.056858 (+ 6760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.056862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.057243 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.057596 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.057606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.059455 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.059463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.062497 (+ 3034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.062509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.063713 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.063722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.064021 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.065363 (+ 1342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.065410 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.069653 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.074167 (+ 4514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.074173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.074187 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075034 (+ 847us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.075530 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.075539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075791 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.076584 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.076589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.077108 (+ 519us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.078440 (+ 1332us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.078744 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":201,"spinlock_wait_cycles":72320,"thread_start_us":448,"threads_started":1}
I20260926 19:27:40.079909 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.040091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:50827)
0926 19:27:40.041082 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.041141 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.050657 (+ 9516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.051939 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.051956 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.053704 (+ 1748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.054821 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.054860 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.061442 (+ 6582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.061448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.063191 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.063206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.064398 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.064404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.064585 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.065433 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.065452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.069139 (+ 3687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.074154 (+ 5015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.074166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.074185 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075067 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.075473 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.075482 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.075787 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.076557 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.076565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.077107 (+ 542us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.078745 (+ 1638us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.078992 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":125,"spinlock_wait_cycles":13952,"thread_start_us":410,"threads_started":1}
I20260926 19:27:40.080389 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.041859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57415 (local address 127.1.39.61:41101)
0926 19:27:40.058565 (+ 16706us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.058570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.058873 (+ 303us) server_negotiation.cc:407] Connection header received
0926 19:27:40.059055 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.059060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.059144 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.059259 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.060421 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.061478 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.063460 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.064113 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.064223 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.071059 (+ 6836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.071067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.071071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.071108 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.074751 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.075501 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.075507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.075510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.075662 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.076091 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.076098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.076105 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.076392 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.076526 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.079774 (+ 3248us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.080020 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16604,"thread_start_us":16707,"threads_started":1}
I20260926 19:27:40.081063 8289 heartbeater.cc:347] Connected to a master server at 127.1.39.60:40345
I20260926 19:27:40.081168 8291 heartbeater.cc:347] Connected to a master server at 127.1.39.61:41101
I20260926 19:27:40.081413 8289 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.082329 8289 heartbeater.cc:511] Master 127.1.39.60:40345 requested a full tablet report, sending...
I20260926 19:27:40.081436 8290 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:40.083123 8290 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.083791 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.043029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.1.39.2:33157)
0926 19:27:40.062631 (+ 19602us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.062640 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.062667 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:27:40.062736 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.062745 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.062984 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.065598 (+ 2614us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.065833 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.067418 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.069492 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.070366 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.071886 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.076720 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.076732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.076751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.076793 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.080584 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.081465 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.081475 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.081478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.081636 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.082171 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.082174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.082175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.082534 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.082727 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.083105 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.083341 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19407,"thread_start_us":20073,"threads_started":1}
I20260926 19:27:40.083989 8290 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
I20260926 19:27:40.081426 8291 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.084764 8291 heartbeater.cc:511] Master 127.1.39.61:41101 requested a full tablet report, sending...
I20260926 19:27:40.084715 8110 ts_manager.cc:194] Registered new tserver with Master: 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623)
I20260926 19:27:40.085132 7322 ts_manager.cc:194] Re-registered known tserver with Master: 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623)
I20260926 19:27:40.086120 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:50827
I20260926 19:27:40.086356 7922 ts_manager.cc:194] Registered new tserver with Master: 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623)
I20260926 19:27:40.086869 7640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:40.172713 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7517
I20260926 19:27:40.189985 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:33157
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:33725
--webserver_port=37133
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:27:40.273262 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: NON_VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: true } } }
I20260926 19:27:40.273689 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:40.275365 8302 raft_consensus.cc:1050] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: attempting to promote NON_VOTER c2064901f9184f989adaca33d7a46085 to VOTER
I20260926 19:27:40.275962 8302 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:40.277848 7931 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 19:27:40.277981 8120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 LEARNER]: Refusing update from remote peer d0b3c26acf0547b9a575729235755a20: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 19:27:40.278463 8302 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 19:27:40.278849 8303 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 19:27:40.282157 8074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2064901f9184f989adaca33d7a46085 (127.1.39.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.283291 7931 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2064901f9184f989adaca33d7a46085 (127.1.39.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.286410 8120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c2064901f9184f989adaca33d7a46085 (127.1.39.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.290629 8149 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.290851 8149 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:40.293274 8308 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.293756 8308 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:40.292552 8307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.294144 8307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:40.294114 8308 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.294319 8308 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:40.294433 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b3c26acf0547b9a575729235755a20" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:40.294655 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20 [sys.catalog]: This master's current role is: LEADER
W20260926 19:27:40.521440 8301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:40.521808 8301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:40.521971 8301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:40.529764 8301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:40.529907 8301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:40.530081 8301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:40.539978 8301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:33157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=37133
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:40.541353 8301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:40.542688 8301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:40.555529 8316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:40.555595 8314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:40.555469 8313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:40.556962 8301 server_base.cc:1034] running on GCE node
I20260926 19:27:40.557663 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:40.558671 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:40.559881 8301 hybrid_clock.cc:648] HybridClock initialized: now 1790450860559823 us; error 52 us; skew 500 ppm
I20260926 19:27:40.560153 8301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:40.563310 8301 init.cc:377] Logged in from keytab as oryx/127.1.39.2@KRBTEST.COM (short username oryx)
I20260926 19:27:40.564638 8301 webserver.cc:460] Webserver started at http://127.1.39.2:37133/ using document root <none> and password file <none>
I20260926 19:27:40.565269 8301 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:40.565446 8301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:40.570551 8301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 19:27:40.573357 8323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:40.574687 8301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:27:40.574848 8301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "f8f324248b154fbe954996cf05f74711"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "e3330a8003bad2e021c893ff90f18d66"
server_key_iv: "a79e97ed3aaca3bd50194114c11a9e23"
server_key_version: "encryptionkey@0"
I20260926 19:27:40.575825 8301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:40.591655 8301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:40.595858 8301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:40.596149 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:40.597172 8301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:40.598601 8301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:40.598717 8301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:40.598845 8301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:40.598944 8301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:40.630956 8301 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:33157
I20260926 19:27:40.631006 8435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:33157 every 8 connection(s)
I20260926 19:27:40.632688 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:40.640542 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8301
I20260926 19:27:40.640851 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7650
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450860, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
Sep 26 19:27:40 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.3@KRBTEST.COM
I20260926 19:27:40.666872 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.635968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:33843 (local address 127.1.39.62:33725)
0926 19:27:40.637146 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.637154 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.642807 (+ 5653us) server_negotiation.cc:407] Connection header received
0926 19:27:40.643086 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.643114 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.643218 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.643316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.646691 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.648166 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.652474 (+ 4308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.653130 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.653242 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.658493 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.658501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.658506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.658568 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.662200 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.663483 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.663490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.663496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.663655 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.664668 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.664676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.664682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.665060 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.665205 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.666208 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.666462 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1043,"thread_start_us":954,"threads_started":1}
I20260926 19:27:40.668048 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.635913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:54103 (local address 127.1.39.61:41101)
0926 19:27:40.650458 (+ 14545us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.650464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.650486 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:27:40.650548 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.650553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.650648 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.651942 (+ 1294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.652149 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.653321 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.655440 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.656022 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.657229 (+ 1207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.660618 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.660624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.660628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.660677 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.664254 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.665333 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.665337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.665340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.665468 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.665926 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.665930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.665933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.666260 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.667362 (+ 1102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.667493 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.667708 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14417,"thread_start_us":14320,"threads_started":1}
I20260926 19:27:40.668920 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.636728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.1.39.2:54103)
0926 19:27:40.638020 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.638087 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.641702 (+ 3615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.650866 (+ 9164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.650871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.651367 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.651666 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.651676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.653493 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.653501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.655278 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.655291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.656234 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.656240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.656471 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.657566 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.657587 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.660397 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.664403 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.664414 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.664418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.665173 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.665590 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.665594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.665597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.665780 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.666635 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.666643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.666775 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.666918 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.667500 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":544,"threads_started":1}
I20260926 19:27:40.669095 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.636728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:33843)
0926 19:27:40.642685 (+ 5957us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.642708 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.642920 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.643888 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.643904 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.645437 (+ 1533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.646428 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.646457 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.648417 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.648426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.652303 (+ 3877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.652324 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.653359 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.653367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.653734 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.654691 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.654725 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.658280 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.662603 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.662613 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.662633 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.663328 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.664207 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.664211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.664215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.664501 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.665272 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.665280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.665744 (+ 464us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.667137 (+ 1393us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.668350 (+ 1213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5759,"mutex_wait_us":1196,"spinlock_wait_cycles":21760,"thread_start_us":6877,"threads_started":1}
I20260926 19:27:40.670711 8437 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:40.670760 8438 heartbeater.cc:347] Connected to a master server at 127.1.39.61:41101
I20260926 19:27:40.671070 8438 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.671074 8437 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.672164 8438 heartbeater.cc:511] Master 127.1.39.61:41101 requested a full tablet report, sending...
I20260926 19:27:40.672187 8437 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
I20260926 19:27:40.673907 7322 ts_manager.cc:194] Re-registered known tserver with Master: f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:40.674008 7922 ts_manager.cc:194] Registered new tserver with Master: f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:40.674371 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.638753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:36831 (local address 127.1.39.60:40345)
0926 19:27:40.649705 (+ 10952us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.649715 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.653035 (+ 3320us) server_negotiation.cc:407] Connection header received
0926 19:27:40.653169 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.653176 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.653252 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.653381 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.654909 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.656061 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.658639 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.659173 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.659302 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.664245 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.664253 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.664260 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.664302 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.668590 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.671027 (+ 2437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.671034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.671040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.671273 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.671749 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.671757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.671762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.672115 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.672346 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.673643 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.673846 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10785,"thread_start_us":10680,"threads_started":1}
I20260926 19:27:40.674801 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.639328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.1.39.2:36831)
0926 19:27:40.652918 (+ 13590us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:40.652939 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:40.653070 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:40.653559 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:40.653563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:40.654229 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:40.654724 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.654737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.656241 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.656246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.658510 (+ 2264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:40.658527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.659537 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.659543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:40.659737 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.660296 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:40.660312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:40.664047 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:40.668749 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.668754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.668758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.670882 (+ 2124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.671410 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:40.671416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:40.671421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:40.671617 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:40.672407 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:40.672416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:40.672573 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 19:27:40.672730 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 19:27:40.673035 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13483,"thread_start_us":13352,"threads_started":1}
I20260926 19:27:40.675002 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:33843
I20260926 19:27:40.675560 8436 heartbeater.cc:347] Connected to a master server at 127.1.39.60:40345
I20260926 19:27:40.675673 8436 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:40.676153 8436 heartbeater.cc:511] Master 127.1.39.60:40345 requested a full tablet report, sending...
I20260926 19:27:40.676887 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:40.641909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.1.39.3:33309)
0926 19:27:40.650893 (+ 8984us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:40.650901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:40.650929 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:27:40.650994 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:40.651003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:40.651243 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:40.651384 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:40.654753 (+ 3369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.656263 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.659275 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:40.660955 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:40.661084 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:40.667309 (+ 6225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:40.667325 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:40.667343 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:40.667384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:40.672115 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.673075 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.673083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.673086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.673240 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:40.673760 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:40.673765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:40.673767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:40.674156 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:40.675834 (+ 1678us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:40.675990 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 19:27:40.676354 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8812,"thread_start_us":8649,"threads_started":1}
I20260926 19:27:40.677703 8110 ts_manager.cc:194] Registered new tserver with Master: f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:40.678776 7773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:40.764297 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7650
I20260926 19:27:40.780606 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:33309
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:33725
--webserver_port=44423
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:27:41.089058 8290 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
W20260926 19:27:41.107658 8449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:41.107959 8449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:41.108057 8449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:41.115418 8449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:41.115514 8449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:41.115629 8449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:41.124915 8449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:33309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=44423
--tserver_master_addrs=127.1.39.62:33725,127.1.39.61:41101,127.1.39.60:40345
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:41.126154 8449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:41.127375 8449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:41.140484 8454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:41.140555 8457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:41.140520 8455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:41.141320 8449 server_base.cc:1034] running on GCE node
I20260926 19:27:41.141937 8449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:41.142913 8449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:41.144106 8449 hybrid_clock.cc:648] HybridClock initialized: now 1790450861144065 us; error 45 us; skew 500 ppm
I20260926 19:27:41.144424 8449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:41 dist-test-slave-9wb0 krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450861, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:41.147995 8449 init.cc:377] Logged in from keytab as oryx/127.1.39.3@KRBTEST.COM (short username oryx)
I20260926 19:27:41.149528 8449 webserver.cc:460] Webserver started at http://127.1.39.3:44423/ using document root <none> and password file <none>
I20260926 19:27:41.150210 8449 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:41.150363 8449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:41.155745 8449 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:27:41.158598 8464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:41.159948 8449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:27:41.160109 8449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "8c1684fc580c4f68b0230019e1f0e6bb"
format_stamp: "Formatted at 2026-09-26 19:27:37 on dist-test-slave-9wb0"
server_key: "08a383b691d5e71c728ba5c6c332eb5f"
server_key_iv: "acec58998a54713d444b434faa0c2043"
server_key_version: "encryptionkey@0"
I20260926 19:27:41.161053 8449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:41.185982 8449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:41.191771 8449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:41.195096 8449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:41.197307 8449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:41.199260 8449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:41.199359 8449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:41.199441 8449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:41.199581 8449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:41.236421 8449 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:33309
I20260926 19:27:41.236518 8576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:33309 every 8 connection(s)
I20260926 19:27:41.238014 8449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:41.241909 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8449
Sep 26 19:27:41 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450861, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
Sep 26 19:27:41 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450861, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.60@KRBTEST.COM
Sep 26 19:27:41 dist-test-slave-9wb0 krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450861, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.61@KRBTEST.COM
I20260926 19:27:41.307986 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.241479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:33101 (local address 127.1.39.62:33725)
0926 19:27:41.260875 (+ 19396us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.260887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.260924 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:27:41.263090 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.263100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.263218 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.263358 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.266032 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.267711 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.270805 (+ 3094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.271563 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.271729 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.295348 (+ 23619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.295354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.295359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.295402 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.301012 (+ 5610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.304698 (+ 3686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.304708 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.304714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.305048 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.306067 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.306072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.306075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.306424 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.306566 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.307415 (+ 849us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.307628 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19200,"thread_start_us":20314,"threads_started":1}
I20260926 19:27:41.308245 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.242088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:33101)
0926 19:27:41.254073 (+ 11985us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.254143 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.262523 (+ 8380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.263710 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.263729 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.264729 (+ 1000us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:41.265814 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.265844 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.267917 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.267925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.270656 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.270665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.271881 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.271892 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.272377 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.279206 (+ 6829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:41.279247 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:41.290171 (+ 10924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:41.301275 (+ 11104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.301282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.301487 (+ 205us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.302246 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.305743 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.305749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.305753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.305914 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.307039 (+ 1125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:41.307042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:41.307164 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.307298 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.307593 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11568,"mutex_wait_us":57,"thread_start_us":18922,"threads_started":1}
I20260926 19:27:41.309690 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.255203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:35765 (local address 127.1.39.60:40345)
0926 19:27:41.272881 (+ 17678us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.272888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.272912 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:27:41.273196 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.273201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.273306 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.273443 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.275136 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.276845 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.279329 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.279993 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.280127 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.287932 (+ 7805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.287942 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.287950 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.288009 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.293670 (+ 5661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.301520 (+ 7850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.301530 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.301536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.301953 (+ 417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.303039 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.303048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.303052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.303538 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.303726 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.308886 (+ 5160us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.309187 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17530,"thread_start_us":17792,"threads_started":1}
I20260926 19:27:41.310066 8578 heartbeater.cc:347] Connected to a master server at 127.1.39.62:33725
I20260926 19:27:41.310534 8578 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:41.311435 8578 heartbeater.cc:511] Master 127.1.39.62:33725 requested a full tablet report, sending...
I20260926 19:27:41.312621 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.261234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.1.39.3:35765)
0926 19:27:41.270509 (+ 9275us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.270533 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.270648 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.273684 (+ 3036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.273692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.274501 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:41.274902 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.274917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.277060 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.277068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.279143 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.279159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.280376 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.280383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.280687 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.281395 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:41.281426 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:41.287586 (+ 6160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:41.293942 (+ 6356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.294055 (+ 113us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.294084 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.300387 (+ 6303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.302121 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.302129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.302135 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.302526 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.303832 (+ 1306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:41.303847 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:41.304444 (+ 597us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.305673 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.306422 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9135,"thread_start_us":9487,"threads_started":1}
I20260926 19:27:41.313095 7322 ts_manager.cc:194] Re-registered known tserver with Master: 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309)
I20260926 19:27:41.313599 8577 heartbeater.cc:347] Connected to a master server at 127.1.39.60:40345
I20260926 19:27:41.313722 8577 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:41.314380 8577 heartbeater.cc:511] Master 127.1.39.60:40345 requested a full tablet report, sending...
I20260926 19:27:41.314411 7322 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:33101
I20260926 19:27:41.315766 8110 ts_manager.cc:194] Registered new tserver with Master: 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309)
I20260926 19:27:41.320847 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.242088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.1.39.3:44241)
0926 19:27:41.261560 (+ 19472us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.261588 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.262527 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.265129 (+ 2602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.265138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.268210 (+ 3072us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:41.268717 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.268736 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.272187 (+ 3451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.272193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.285365 (+ 13172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.285419 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.288427 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.288436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.288710 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.290523 (+ 1813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:41.290557 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:41.303782 (+ 13225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:41.318240 (+ 14458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.318243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.318246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.318621 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.319137 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:41.319142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:41.319146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:41.319288 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:41.319788 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:41.319792 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:41.319951 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.320097 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.320405 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19260,"mutex_wait_us":108,"spinlock_wait_cycles":36736,"thread_start_us":11262,"threads_started":1}
I20260926 19:27:41.321796 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.244053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:44241 (local address 127.1.39.61:41101)
0926 19:27:41.264351 (+ 20298us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.264360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.264391 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:27:41.264475 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.264482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.264644 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.264857 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.269086 (+ 4229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.271966 (+ 2880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.285779 (+ 13813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.287849 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.287999 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.314054 (+ 26055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.314062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.314067 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.314170 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.318084 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.318771 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.318781 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.318787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.318980 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.319421 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.319426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.319429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.319670 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.319825 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.321132 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.321380 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20115,"thread_start_us":19991,"threads_started":1}
I20260926 19:27:41.322597 8579 heartbeater.cc:347] Connected to a master server at 127.1.39.61:41101
I20260926 19:27:41.322724 8579 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:41.323112 8579 heartbeater.cc:511] Master 127.1.39.61:41101 requested a full tablet report, sending...
I20260926 19:27:41.324215 7922 ts_manager.cc:194] Registered new tserver with Master: 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309)
I20260926 19:27:41.333575 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.273134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.1.39.62:33725)
0926 19:27:41.280964 (+ 7830us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.280970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.295036 (+ 14066us) server_negotiation.cc:407] Connection header received
0926 19:27:41.295258 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.295266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.295348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.295445 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.310682 (+ 15237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.311778 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.320621 (+ 8843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.321143 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.321264 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.325609 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.325617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.325622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.325657 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.330241 (+ 4584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.331183 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.331192 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.331197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.331360 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.331820 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.331826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.331830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.332206 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.332478 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.332840 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.333071 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7682,"thread_start_us":7624,"threads_started":1}
I20260926 19:27:41.336383 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.274751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.1.39.61:41101)
0926 19:27:41.302813 (+ 28062us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.302823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.306556 (+ 3733us) server_negotiation.cc:407] Connection header received
0926 19:27:41.306628 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.306632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.306701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.306817 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.319913 (+ 13096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.321082 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.323552 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.324153 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.324258 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.329779 (+ 5521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.329787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.329792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.329833 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.333073 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.333929 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.333934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.333939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.334086 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.334570 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.334577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.334581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.334942 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.335153 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.335725 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.335941 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27887,"thread_start_us":27731,"threads_started":1}
I20260926 19:27:41.340339 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.295040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.1.39.60:40345)
0926 19:27:41.311185 (+ 16145us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.311194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.319422 (+ 8228us) server_negotiation.cc:407] Connection header received
0926 19:27:41.320116 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.320122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.320226 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.320290 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.324247 (+ 3957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.325410 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.327570 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.327996 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.328142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.332661 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.332690 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.332696 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.332739 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.337100 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.337899 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.337903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.337905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.338082 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.338475 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.338484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.338486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.338733 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.339026 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.339819 (+ 793us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.340029 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16006,"thread_start_us":19348,"threads_started":1}
I20260926 19:27:41.338332 7322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:41.346216 7322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:41.383873 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.367490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.1.39.2:33157)
0926 19:27:41.368349 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.368363 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.374855 (+ 6492us) server_negotiation.cc:407] Connection header received
0926 19:27:41.375413 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.375421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.375667 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.375854 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:41.376915 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.378280 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.380398 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.381982 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.382088 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.382308 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.382971 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.383162 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":424,"threads_started":1}
I20260926 19:27:41.383939 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.366189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33623 (local address 127.0.0.1:50290)
0926 19:27:41.373941 (+ 7752us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.373967 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.374163 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.375519 (+ 1356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.375527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.375561 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:41.376170 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.376196 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.378312 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.378318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.380768 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.380786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.382900 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.382907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.383195 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.383242 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.383367 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.383532 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7574,"mutex_wait_us":48,"thread_start_us":8258,"threads_started":1}
I20260926 19:27:41.384315 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.366640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.1.39.1:33623)
0926 19:27:41.371797 (+ 5157us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.371808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.374135 (+ 2327us) server_negotiation.cc:407] Connection header received
0926 19:27:41.374378 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.374390 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.374799 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.375152 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:41.376481 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.378126 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.380997 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.382667 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.382799 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.383010 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.383434 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.384051 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2937,"thread_start_us":2824,"threads_started":1}
I20260926 19:27:41.385128 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.366826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:33157 (local address 127.0.0.1:59094)
0926 19:27:41.374701 (+ 7875us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.374714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.374962 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.376214 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.376218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.376249 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:41.376682 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.376695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.378429 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.378435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.380207 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.380225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.382528 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.382534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.382756 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.382794 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.384664 (+ 1870us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.384800 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7749,"mutex_wait_us":68,"thread_start_us":6739,"threads_started":1}
I20260926 19:27:41.386317 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.365584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:33309 (local address 127.0.0.1:52156)
0926 19:27:41.366151 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.366171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.366290 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.377979 (+ 11689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.377984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.378007 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:41.378439 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.378450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.381464 (+ 3014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.381472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.383318 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.383334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.385566 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.385572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.385713 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.385736 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.385792 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.385933 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":1,"thread_start_us":289,"threads_started":1}
I20260926 19:27:41.387588 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.365991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.1.39.3:33309)
0926 19:27:41.377407 (+ 11416us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.377416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.377446 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:27:41.377534 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.377541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.377785 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.378659 (+ 874us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:41.378945 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.380542 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.383543 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.385159 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.386679 (+ 1520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.386896 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.386966 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.387147 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11164,"thread_start_us":10991,"threads_started":1}
I20260926 19:27:41.388180 8224 tablet_service.cc:1467] Processing CreateTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (DEFAULT_TABLE table=test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:41.388934 8371 tablet_service.cc:1467] Processing CreateTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (DEFAULT_TABLE table=test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:41.389822 8224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:41.390308 8371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:41.391148 8512 tablet_service.cc:1467] Processing CreateTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (DEFAULT_TABLE table=test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:41.392437 8512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:41.401948 8614 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Bootstrap starting.
I20260926 19:27:41.405903 8614 tablet_bootstrap.cc:654] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:41.407238 8614 log.cc:824] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:41.409484 8615 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Bootstrap starting.
I20260926 19:27:41.410249 8614 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: No bootstrap required, opened a new log
I20260926 19:27:41.410511 8614 ts_tablet_manager.cc:1397] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260926 19:27:41.412912 8615 tablet_bootstrap.cc:654] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:41.414263 8617 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Bootstrap starting.
I20260926 19:27:41.414315 8615 log.cc:824] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:41.417429 8615 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: No bootstrap required, opened a new log
I20260926 19:27:41.417693 8615 ts_tablet_manager.cc:1397] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260926 19:27:41.418040 8617 tablet_bootstrap.cc:654] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:41.418994 8617 log.cc:824] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:41.421370 8617 tablet_bootstrap.cc:492] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: No bootstrap required, opened a new log
I20260926 19:27:41.421617 8617 ts_tablet_manager.cc:1397] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260926 19:27:41.421739 8614 raft_consensus.cc:348] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.422216 8614 raft_consensus.cc:374] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:41.422322 8614 raft_consensus.cc:729] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f324248b154fbe954996cf05f74711, State: Initialized, Role: FOLLOWER
I20260926 19:27:41.422922 8614 consensus_queue.cc:260] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.425648 8437 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
I20260926 19:27:41.428308 8615 raft_consensus.cc:348] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.428733 8615 raft_consensus.cc:374] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:41.428833 8615 raft_consensus.cc:729] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827c49b220cf4c4a98ae0a3c38f06420, State: Initialized, Role: FOLLOWER
I20260926 19:27:41.429565 8615 consensus_queue.cc:260] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.430081 8614 ts_tablet_manager.cc:1428] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20260926 19:27:41.431667 8615 ts_tablet_manager.cc:1428] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260926 19:27:41.436710 8617 raft_consensus.cc:348] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.437036 8617 raft_consensus.cc:374] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:41.437109 8617 raft_consensus.cc:729] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1684fc580c4f68b0230019e1f0e6bb, State: Initialized, Role: FOLLOWER
I20260926 19:27:41.437690 8617 consensus_queue.cc:260] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.438758 8578 heartbeater.cc:503] Master 127.1.39.62:33725 was elected leader, sending a full tablet report...
I20260926 19:27:41.439334 8617 ts_tablet_manager.cc:1428] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Time spent starting tablet: real 0.018s user 0.008s sys 0.003s
I20260926 19:27:41.469535 8622 raft_consensus.cc:482] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:41.469866 8622 raft_consensus.cc:504] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.471540 8622 leader_election.cc:288] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623), f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:41.477142 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.471838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:52517 (local address 127.1.39.1:33623)
0926 19:27:41.472032 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.472038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.472059 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:41.472256 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.472260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.472330 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.472437 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:41.472826 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.473591 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.475429 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.476249 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.476397 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.476495 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.476783 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.476908 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:27:41.477198 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.471733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33623 (local address 127.1.39.3:52517)
0926 19:27:41.471968 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.471989 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.472101 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.472457 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.472462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.472499 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:41.472678 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.472686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.473784 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.473788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.475242 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.475275 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.476409 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.476414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.476622 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.476655 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.476758 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.476901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":72}
I20260926 19:27:41.477190 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.472128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:57925 (local address 127.1.39.2:33157)
0926 19:27:41.472330 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.472334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.472348 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:27:41.472400 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.472403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.472454 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.472564 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:41.472940 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.473681 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.475344 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.476251 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.476423 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.476512 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.476833 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.476950 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 19:27:41.477243 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.471766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:33157 (local address 127.1.39.3:57925)
0926 19:27:41.471976 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:41.471989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:41.472104 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:41.472573 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:41.472578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:41.472601 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:41.472794 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.472802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.473809 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.473816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.475170 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:41.475181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.476469 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.476475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:41.476683 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.476705 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:27:41.476810 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 19:27:41.476939 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":5,"spinlock_wait_cycles":2432}
I20260926 19:27:41.478308 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5013d71f54da4a139c3cdf1ca6fdcdd7" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827c49b220cf4c4a98ae0a3c38f06420" is_pre_election: true
I20260926 19:27:41.478304 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5013d71f54da4a139c3cdf1ca6fdcdd7" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f324248b154fbe954996cf05f74711" is_pre_election: true
I20260926 19:27:41.478768 8391 raft_consensus.cc:2454] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1684fc580c4f68b0230019e1f0e6bb in term 0.
I20260926 19:27:41.478777 8244 raft_consensus.cc:2454] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1684fc580c4f68b0230019e1f0e6bb in term 0.
I20260926 19:27:41.479379 8467 leader_election.cc:302] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 827c49b220cf4c4a98ae0a3c38f06420, 8c1684fc580c4f68b0230019e1f0e6bb, f8f324248b154fbe954996cf05f74711; no voters:
I20260926 19:27:41.479828 8622 raft_consensus.cc:2781] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:41.479975 8622 raft_consensus.cc:482] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:41.480088 8622 raft_consensus.cc:3037] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:41.482453 8622 raft_consensus.cc:504] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
I20260926 19:27:41.483021 8622 leader_election.cc:288] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 election: Requested vote from peers 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623), f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:41.483451 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5013d71f54da4a139c3cdf1ca6fdcdd7" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827c49b220cf4c4a98ae0a3c38f06420"
I20260926 19:27:41.483482 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5013d71f54da4a139c3cdf1ca6fdcdd7" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f324248b154fbe954996cf05f74711"
I20260926 19:27:41.483669 8391 raft_consensus.cc:3037] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:41.483670 8244 raft_consensus.cc:3037] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:41.487308 8244 raft_consensus.cc:2454] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1684fc580c4f68b0230019e1f0e6bb in term 1.
I20260926 19:27:41.487308 8391 raft_consensus.cc:2454] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1684fc580c4f68b0230019e1f0e6bb in term 1.
I20260926 19:27:41.487728 8466 leader_election.cc:302] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827c49b220cf4c4a98ae0a3c38f06420, 8c1684fc580c4f68b0230019e1f0e6bb; no voters:
I20260926 19:27:41.488086 8622 raft_consensus.cc:2781] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:41.488564 8622 raft_consensus.cc:686] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 LEADER]: Becoming Leader. State: Replica: 8c1684fc580c4f68b0230019e1f0e6bb, State: Running, Role: LEADER
I20260926 19:27:41.489127 8622 consensus_queue.cc:237] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } }
W20260926 19:27:41.490859 8580 tablet.cc:2369] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:41.494431 7322 catalog_manager.cc:5495] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3). New cstate: current_term: 1 leader_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:41.532114 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.526851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.1.39.3:33309)
0926 19:27:41.527059 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.527065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.527084 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:41.527308 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.527312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.527392 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.527553 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:41.527905 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.528626 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.530003 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.530419 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.530559 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.531008 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:41.531539 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:41.531714 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.531765 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.531846 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 19:27:41.540401 8292 tablet.cc:2369] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:41.540758 8391 raft_consensus.cc:1261] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Refusing update from remote peer 8c1684fc580c4f68b0230019e1f0e6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:41.540761 8244 raft_consensus.cc:1261] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Refusing update from remote peer 8c1684fc580c4f68b0230019e1f0e6bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:41.541896 8622 consensus_queue.cc:1035] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:41.542279 8623 consensus_queue.cc:1035] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:41.557454 8626 mvcc.cc:205] Tried to move back new op lower bound from 7333686728859066368 to 7333686728663875584. Current Snapshot: MvccSnapshot[applied={T|T < 7333686728859066368}]
I20260926 19:27:41.558290 8628 mvcc.cc:205] Tried to move back new op lower bound from 7333686728859066368 to 7333686728663875584. Current Snapshot: MvccSnapshot[applied={T|T < 7333686728859066368}]
I20260926 19:27:41.572835 7322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:41.573115 7322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:41.580741 7322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0b3c26acf0547b9a575729235755a20: Sending DeleteTablet for 3 replicas of tablet 5013d71f54da4a139c3cdf1ca6fdcdd7
I20260926 19:27:41.581964 8512 tablet_service.cc:1514] Processing DeleteTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:41 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:52156
I20260926 19:27:41.582086 8371 tablet_service.cc:1514] Processing DeleteTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:41 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59094
I20260926 19:27:41.582903 8224 tablet_service.cc:1514] Processing DeleteTablet for tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:41 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:50290
I20260926 19:27:41.590098 8637 tablet_replica.cc:330] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: stopping tablet replica
I20260926 19:27:41.590687 8637 raft_consensus.cc:2229] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:41.591274 8637 raft_consensus.cc:2258] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:41.592101 8635 tablet_replica.cc:330] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: stopping tablet replica
I20260926 19:27:41.592484 8635 raft_consensus.cc:2229] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:41.593134 8635 raft_consensus.cc:2258] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:41.594512 8637 ts_tablet_manager.cc:1905] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:41.595191 8636 tablet_replica.cc:330] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: stopping tablet replica
I20260926 19:27:41.595574 8636 raft_consensus.cc:2229] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:41.596050 8636 raft_consensus.cc:2258] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:41.596311 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7290
I20260926 19:27:41.597959 7352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:41.598104 8636 ts_tablet_manager.cc:1905] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:41.601915 8637 ts_tablet_manager.cc:1918] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:41.602134 8637 log.cc:1196] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/5013d71f54da4a139c3cdf1ca6fdcdd7
I20260926 19:27:41.602735 8637 ts_tablet_manager.cc:1939] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting consensus metadata
I20260926 19:27:41.604492 7307 catalog_manager.cc:4841] TS 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623): tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (table test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]) successfully deleted
I20260926 19:27:41.605075 8636 ts_tablet_manager.cc:1918] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:41.605245 8636 log.cc:1196] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/5013d71f54da4a139c3cdf1ca6fdcdd7
I20260926 19:27:41.606194 8636 ts_tablet_manager.cc:1939] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P f8f324248b154fbe954996cf05f74711: Deleting consensus metadata
I20260926 19:27:41.606712 8635 ts_tablet_manager.cc:1905] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:41.615178 8635 ts_tablet_manager.cc:1918] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:41.615374 8635 log.cc:1196] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/5013d71f54da4a139c3cdf1ca6fdcdd7
I20260926 19:27:41.615837 8635 ts_tablet_manager.cc:1939] T 5013d71f54da4a139c3cdf1ca6fdcdd7 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting consensus metadata
I20260926 19:27:41.626431 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.585721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.1.39.61:41101)
0926 19:27:41.585929 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.585936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.594588 (+ 8652us) server_negotiation.cc:407] Connection header received
0926 19:27:41.595155 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.595161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.595246 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.595354 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.608255 (+ 12901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.609755 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.612298 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.613233 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.613394 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.620650 (+ 7256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.620655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.620658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.620690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.623794 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.624522 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.624530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.624538 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.624648 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.625012 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.625016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.625019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.625273 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.625564 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.625909 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.626117 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:27:41.641629 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.594161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.1.39.60:40345)
0926 19:27:41.594329 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.594336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.626467 (+ 32131us) server_negotiation.cc:407] Connection header received
0926 19:27:41.626718 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.626725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.626808 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.626878 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.628758 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.629555 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.631193 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.631816 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.631905 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.635260 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.635266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.635271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.635309 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.638688 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.639428 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.639433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.639437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.639574 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.639959 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.639964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.639968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.640277 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.640489 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.641022 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.641250 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:27:41.787658 7307 catalog_manager.cc:4841] TS 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309): tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (table test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]) successfully deleted
I20260926 19:27:41.788139 7308 catalog_manager.cc:4841] TS f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157): tablet 5013d71f54da4a139c3cdf1ca6fdcdd7 (table test-table [id=b8d3fcdf6c3b423fafdd8ea036cf940b]) successfully deleted
W20260926 19:27:41.788317 7308 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.1.39.2:33157, user_credentials={real_user=slave}} blocked reactor thread for 180739us
I20260926 19:27:41.788511 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7290
I20260926 19:27:41.832592 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.816090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.1.39.61:41101)
0926 19:27:41.816303 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.816310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.816956 (+ 646us) server_negotiation.cc:407] Connection header received
0926 19:27:41.817223 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.817229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.817347 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.817426 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.819717 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.820457 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.822288 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.822841 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.822995 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.826349 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.826356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.826360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.826400 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.829772 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.830571 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.830578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.830584 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.830727 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.831135 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.831140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.831143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.831503 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.831773 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.832091 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.832282 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:27:41.833778 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:41.817778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.1.39.60:40345)
0926 19:27:41.817980 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:41.817988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:41.818413 (+ 425us) server_negotiation.cc:407] Connection header received
0926 19:27:41.818549 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:41.818554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:41.818635 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:41.818736 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:41.820709 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.821871 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.823736 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:41.824160 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:41.824252 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:41.827427 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:41.827433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:41.827438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:41.827475 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:41.830769 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.831478 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.831484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.831487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.831635 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:41.832094 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:41.832100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:41.832104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:41.832447 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:41.832648 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:41.833223 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 19:27:41.833457 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:27:42.565734 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:42.564813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:38923)
0926 19:27:42.565356 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:42.565498 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":240,"threads_started":1}
I20260926 19:27:42.565974 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:42.564685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:48403)
0926 19:27:42.565550 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:42.565742 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":664,"threads_started":1}
W20260926 19:27:42.566159 8437 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:33725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
W20260926 19:27:42.566352 8578 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:33725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:42.570909 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:42.570087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:49949)
0926 19:27:42.570575 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:42.570701 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":249,"threads_started":1}
W20260926 19:27:42.571300 8290 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:33725 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.085556 8649 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0b3c26acf0547b9a575729235755a20)
I20260926 19:27:43.085785 8649 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:43.087129 8649 leader_election.cc:288] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61:41101)
I20260926 19:27:43.087759 8650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d0b3c26acf0547b9a575729235755a20)
I20260926 19:27:43.087998 8650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:43.095165 8650 leader_election.cc:288] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), c2064901f9184f989adaca33d7a46085 (127.1.39.60:40345)
I20260926 19:27:43.100999 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.087273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56134)
0926 19:27:43.100392 (+ 13119us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.100626 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13000,"thread_start_us":12858,"threads_started":1}
I20260926 19:27:43.102133 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.089174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56140)
0926 19:27:43.101698 (+ 12524us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.101851 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12428,"thread_start_us":12648,"threads_started":1}
W20260926 19:27:43.102420 7908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
W20260926 19:27:43.103521 8096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.104123 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.103687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56146)
0926 19:27:43.103888 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.103969 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 19:27:43.104346 7908 leader_election.cc:334] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.107105 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.106817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56160)
0926 19:27:43.106939 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.106996 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 19:27:43.107376 8096 leader_election.cc:334] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.110683 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.099844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:40345 (local address 127.0.0.1:50392)
0926 19:27:43.102524 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.102541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:43.102668 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:43.103867 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:43.103871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:43.103895 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:43.104306 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:43.104320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.106218 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.106223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:43.108170 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:43.108184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.109745 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.109752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:43.109988 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:43.110040 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 19:27:43.110125 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 19:27:43.110256 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2525,"mutex_wait_us":40,"thread_start_us":2410,"threads_started":1}
I20260926 19:27:43.110731 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.087331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:41101 (local address 127.0.0.1:46368)
0926 19:27:43.101946 (+ 14615us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.101967 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:43.102153 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:43.102874 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:43.102883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:43.102921 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:43.103341 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:43.103351 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.105973 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.105979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:43.108135 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:43.108151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.109745 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.109752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:43.109943 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:43.109980 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:27:43.110081 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:27:43.110248 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14471,"mutex_wait_us":1,"thread_start_us":14303,"threads_started":1}
I20260926 19:27:43.110759 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.100388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.1.39.60:40345)
0926 19:27:43.103563 (+ 3175us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:43.103570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:43.103586 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:27:43.103643 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:43.103648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:43.103715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:43.104481 (+ 766us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:43.104778 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.106057 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.108319 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.109570 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.109695 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:43.109825 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:43.110165 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 19:27:43.110332 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":3051,"thread_start_us":2915,"threads_started":1}
I20260926 19:27:43.110709 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.087714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.1.39.61:41101)
0926 19:27:43.099580 (+ 11866us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:43.099589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:43.102273 (+ 2684us) server_negotiation.cc:407] Connection header received
0926 19:27:43.102358 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:43.102364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:43.102457 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:43.102578 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:43.103555 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.105810 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.108308 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:43.109580 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:43.109711 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:43.109859 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:43.110140 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 19:27:43.110284 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":11689,"thread_start_us":11450,"threads_started":1}
I20260926 19:27:43.111305 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2064901f9184f989adaca33d7a46085" is_pre_election: true
I20260926 19:27:43.111639 8120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49931b8ce6234c9fbaf8dcd18c3aeb99 in term 1.
I20260926 19:27:43.111635 7931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2064901f9184f989adaca33d7a46085" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" is_pre_election: true
I20260926 19:27:43.111938 7931 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2064901f9184f989adaca33d7a46085 in term 1.
I20260926 19:27:43.112150 7907 leader_election.cc:302] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99, c2064901f9184f989adaca33d7a46085; no voters: d0b3c26acf0547b9a575729235755a20
I20260926 19:27:43.112430 8650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 19:27:43.112550 8650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0b3c26acf0547b9a575729235755a20)
I20260926 19:27:43.112661 8650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:27:43.112668 8095 leader_election.cc:302] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99, c2064901f9184f989adaca33d7a46085; no voters: d0b3c26acf0547b9a575729235755a20
I20260926 19:27:43.113056 8649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 19:27:43.113198 8649 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d0b3c26acf0547b9a575729235755a20)
I20260926 19:27:43.113322 8649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:27:43.115345 8650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:43.115394 8649 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:43.115903 8650 leader_election.cc:288] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 election: Requested vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), c2064901f9184f989adaca33d7a46085 (127.1.39.60:40345)
I20260926 19:27:43.116039 8649 leader_election.cc:288] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 election: Requested vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), 49931b8ce6234c9fbaf8dcd18c3aeb99 (127.1.39.61:41101)
I20260926 19:27:43.116515 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.115990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56168)
0926 19:27:43.116271 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.116362 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 19:27:43.116536 7931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c2064901f9184f989adaca33d7a46085" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99"
I20260926 19:27:43.116820 7931 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c2064901f9184f989adaca33d7a46085 in current term 2: Already voted for candidate 49931b8ce6234c9fbaf8dcd18c3aeb99 in this term.
I20260926 19:27:43.117146 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2064901f9184f989adaca33d7a46085"
I20260926 19:27:43.117357 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.116434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56170)
0926 19:27:43.116602 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.117207 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 19:27:43.117442 8120 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 49931b8ce6234c9fbaf8dcd18c3aeb99 in current term 2: Already voted for candidate c2064901f9184f989adaca33d7a46085 in this term.
I20260926 19:27:43.117611 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.117072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56186)
0926 19:27:43.117385 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.117475 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
W20260926 19:27:43.117772 7908 leader_election.cc:334] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.117929 7907 leader_election.cc:302] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99; no voters: c2064901f9184f989adaca33d7a46085, d0b3c26acf0547b9a575729235755a20
I20260926 19:27:43.118294 8650 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 19:27:43.118415 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.118099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56202)
0926 19:27:43.118214 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.118279 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 19:27:43.118566 8096 leader_election.cc:334] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:43.118660 8096 leader_election.cc:302] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c2064901f9184f989adaca33d7a46085; no voters: 49931b8ce6234c9fbaf8dcd18c3aeb99, d0b3c26acf0547b9a575729235755a20
I20260926 19:27:43.118856 8649 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 19:27:43.567977 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.567095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:60763)
0926 19:27:43.567664 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.567762 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":319,"threads_started":1}
I20260926 19:27:43.568221 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.567193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:55657)
0926 19:27:43.567935 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.568012 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":558,"threads_started":1}
I20260926 19:27:43.572914 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:43.571932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:54359)
0926 19:27:43.572388 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:43.572709 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":259,"threads_started":1}
I20260926 19:27:44.569777 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.569031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:54717)
0926 19:27:44.569511 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.569622 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":380,"threads_started":1}
I20260926 19:27:44.569799 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.568914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:48021)
0926 19:27:44.569454 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.569574 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260926 19:27:44.574452 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.573723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:50253)
0926 19:27:44.574167 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.574271 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":356,"threads_started":1}
I20260926 19:27:44.659020 8665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:44.659263 8665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:44.659895 8665 leader_election.cc:288] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), c2064901f9184f989adaca33d7a46085 (127.1.39.60:40345)
I20260926 19:27:44.660435 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2064901f9184f989adaca33d7a46085" is_pre_election: true
I20260926 19:27:44.660679 8120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49931b8ce6234c9fbaf8dcd18c3aeb99 in term 2.
I20260926 19:27:44.660998 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.660158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56414)
0926 19:27:44.660662 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.660757 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":255,"threads_started":1}
I20260926 19:27:44.661164 7907 leader_election.cc:302] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99, c2064901f9184f989adaca33d7a46085; no voters:
I20260926 19:27:44.661504 8665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 19:27:44.661667 8665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:44.661799 8665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:27:44.662447 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.662132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56420)
0926 19:27:44.662268 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.662334 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 19:27:44.662631 7908 leader_election.cc:334] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:44.664252 8665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:44.664816 8665 leader_election.cc:288] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 election: Requested vote from peers d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725), c2064901f9184f989adaca33d7a46085 (127.1.39.60:40345)
I20260926 19:27:44.665249 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.664896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56432)
0926 19:27:44.665070 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.665133 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 19:27:44.665392 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c2064901f9184f989adaca33d7a46085"
I20260926 19:27:44.665647 8120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:27:44.666550 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.666199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56436)
0926 19:27:44.666355 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.666437 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 19:27:44.666734 7908 leader_election.cc:334] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
I20260926 19:27:44.667853 8120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49931b8ce6234c9fbaf8dcd18c3aeb99 in term 3.
I20260926 19:27:44.668252 7907 leader_election.cc:302] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49931b8ce6234c9fbaf8dcd18c3aeb99, c2064901f9184f989adaca33d7a46085; no voters: d0b3c26acf0547b9a575729235755a20
I20260926 19:27:44.668548 8665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 19:27:44.668974 8665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [term 3 LEADER]: Becoming Leader. State: Replica: 49931b8ce6234c9fbaf8dcd18c3aeb99, State: Running, Role: LEADER
I20260926 19:27:44.669425 8665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } }
I20260926 19:27:44.671723 8668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49931b8ce6234c9fbaf8dcd18c3aeb99. Latest consensus state: current_term: 3 leader_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:44.671976 8668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:44.672713 8670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:44.676767 8670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:44.677644 8670 catalog_manager.cc:1259] Loaded cluster ID: 0293c0558520431e87dd0a0ffc992310
I20260926 19:27:44.677719 8670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:44.678903 8670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:44.679811 8670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Loaded TSK: 0
I20260926 19:27:44.680930 8670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:27:44.701948 7921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:44.709249 7921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:44.713907 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.713456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56450)
0926 19:27:44.713652 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.713764 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 19:27:44.713932 8120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [term 3 FOLLOWER]: Refusing update from remote peer 49931b8ce6234c9fbaf8dcd18c3aeb99: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 19:27:44.714669 8668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 19:27:44.715062 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56454)
0926 19:27:44.714783 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.714899 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:27:44.715234 7908 consensus_peers.cc:487] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 -> Peer d0b3c26acf0547b9a575729235755a20 (127.1.39.62:33725): Couldn't send request to peer d0b3c26acf0547b9a575729235755a20. Status: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 19:27:44.717794 8672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49931b8ce6234c9fbaf8dcd18c3aeb99. Latest consensus state: current_term: 3 leader_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:44.718096 8672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:44.719390 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:44.719820 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:44.722366 8673 mvcc.cc:205] Tried to move back new op lower bound from 7333686741861134336 to 7333686741689143296. Current Snapshot: MvccSnapshot[applied={T|T < 7333686741861134336}]
I20260926 19:27:44.729744 8672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d0b3c26acf0547b9a575729235755a20" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33725 } } peers { permanent_uuid: "49931b8ce6234c9fbaf8dcd18c3aeb99" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 41101 } attrs { promote: false } } peers { permanent_uuid: "c2064901f9184f989adaca33d7a46085" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 40345 } attrs { promote: false } } }
I20260926 19:27:44.730002 8672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2064901f9184f989adaca33d7a46085 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:27:44.744254 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.734488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:33309 (local address 127.0.0.1:52174)
0926 19:27:44.734659 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.734680 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:44.734817 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:44.735485 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:44.735489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:44.735513 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:44.735873 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.735889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.737813 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.737822 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.740421 (+ 2599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.740443 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.743208 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.743218 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.743483 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.743541 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 19:27:44.743628 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 19:27:44.743816 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 19:27:44.745817 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.734596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.1.39.3:33309)
0926 19:27:44.735143 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:44.735149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:44.735167 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:27:44.735235 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:44.735240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:44.735327 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:44.736040 (+ 713us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:44.736288 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.737606 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.740732 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.742967 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.744965 (+ 1998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.745085 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:44.745147 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 19:27:44.745354 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
I20260926 19:27:44.747398 8512 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb2b58c73404d9e820298fc097659 (DEFAULT_TABLE table=test-table [id=23b61aaa92394089b9513b6493777c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:44.747893 8512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb2b58c73404d9e820298fc097659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:44.754387 8666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.735202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33623 (local address 127.0.0.1:50306)
0926 19:27:44.744553 (+ 9351us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.744577 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:44.744701 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:44.747872 (+ 3171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:44.747879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:44.747910 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:44.748081 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.748093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.749521 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.749527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.751783 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.751799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.753467 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.753474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.753660 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.753701 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 19:27:44.753772 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 19:27:44.753958 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9213,"mutex_wait_us":1,"thread_start_us":11106,"threads_started":1}
I20260926 19:27:44.755483 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.735591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.1.39.2:33157)
0926 19:27:44.746675 (+ 11084us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:44.746684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:44.746703 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:44.747657 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:44.747662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:44.747731 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:44.747807 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:44.748860 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.750199 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.752565 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.753639 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.753775 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.753895 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:44.754893 (+ 998us) server_negotiation.cc:299] Negotiation successful
0926 19:27:44.755060 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10944,"thread_start_us":11111,"threads_started":1}
I20260926 19:27:44.755612 8683 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Bootstrap starting.
I20260926 19:27:44.755889 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.735463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:33157 (local address 127.0.0.1:59106)
0926 19:27:44.746613 (+ 11150us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.746632 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:44.746755 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:44.748210 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:44.748216 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:44.748256 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:44.748667 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.748676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.750372 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.750381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.752385 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.752398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.754037 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.754044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.754249 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.754283 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:27:44.754394 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 19:27:44.754555 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10999,"thread_start_us":13372,"threads_started":1}
I20260926 19:27:44.756227 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.735582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.1.39.1:33623)
0926 19:27:44.747547 (+ 11965us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:44.747556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:44.747577 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:44.747648 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:44.747653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:44.747734 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:44.747833 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:44.748257 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.749375 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.752024 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.753261 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.755068 (+ 1807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.755243 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:44.755293 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 19:27:44.755438 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11817,"thread_start_us":11690,"threads_started":1}
I20260926 19:27:44.756919 8224 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb2b58c73404d9e820298fc097659 (DEFAULT_TABLE table=test-table [id=23b61aaa92394089b9513b6493777c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:44.756919 8371 tablet_service.cc:1467] Processing CreateTablet for tablet 217eb2b58c73404d9e820298fc097659 (DEFAULT_TABLE table=test-table [id=23b61aaa92394089b9513b6493777c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:44.757395 8371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb2b58c73404d9e820298fc097659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:44.757395 8224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 217eb2b58c73404d9e820298fc097659. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:44.757726 8683 tablet_bootstrap.cc:654] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:44.760842 8683 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: No bootstrap required, opened a new log
I20260926 19:27:44.761018 8683 ts_tablet_manager.cc:1397] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 19:27:44.761724 8683 raft_consensus.cc:348] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.761927 8683 raft_consensus.cc:374] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:44.762039 8683 raft_consensus.cc:729] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1684fc580c4f68b0230019e1f0e6bb, State: Initialized, Role: FOLLOWER
I20260926 19:27:44.762588 8683 consensus_queue.cc:260] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.763484 8683 ts_tablet_manager.cc:1428] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 19:27:44.764448 8688 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Bootstrap starting.
I20260926 19:27:44.764919 8686 raft_consensus.cc:482] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:44.765244 8686 raft_consensus.cc:504] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.766330 8688 tablet_bootstrap.cc:654] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:44.767555 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.767067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:59071)
0926 19:27:44.767283 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.767407 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:27:44.767654 8687 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Bootstrap starting.
I20260926 19:27:44.769237 8687 tablet_bootstrap.cc:654] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:44.770519 8686 leader_election.cc:288] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157), 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623)
I20260926 19:27:44.770541 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f324248b154fbe954996cf05f74711" is_pre_election: true
I20260926 19:27:44.770602 8244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "827c49b220cf4c4a98ae0a3c38f06420" is_pre_election: true
I20260926 19:27:44.770817 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.768928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:57935)
0926 19:27:44.770576 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.770660 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1523}
I20260926 19:27:44.771512 8579 heartbeater.cc:503] Master 127.1.39.61:41101 was elected leader, sending a full tablet report...
W20260926 19:27:44.771508 8466 leader_election.cc:341] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623): Illegal state: must be running to vote when last-logged opid is not known
W20260926 19:27:44.771744 8467 leader_election.cc:341] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157): Illegal state: must be running to vote when last-logged opid is not known
I20260926 19:27:44.771880 8467 leader_election.cc:302] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c1684fc580c4f68b0230019e1f0e6bb; no voters: 827c49b220cf4c4a98ae0a3c38f06420, f8f324248b154fbe954996cf05f74711
I20260926 19:27:44.772491 8688 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: No bootstrap required, opened a new log
I20260926 19:27:44.772663 8686 raft_consensus.cc:2726] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:27:44.772985 8688 ts_tablet_manager.cc:1397] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260926 19:27:44.773278 8687 tablet_bootstrap.cc:492] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: No bootstrap required, opened a new log
I20260926 19:27:44.773715 8687 ts_tablet_manager.cc:1397] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 19:27:44.773937 8688 raft_consensus.cc:348] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.774140 8688 raft_consensus.cc:374] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:44.774235 8688 raft_consensus.cc:729] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827c49b220cf4c4a98ae0a3c38f06420, State: Initialized, Role: FOLLOWER
I20260926 19:27:44.774513 8688 consensus_queue.cc:260] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.774514 8687 raft_consensus.cc:348] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.774712 8687 raft_consensus.cc:374] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:44.774818 8687 raft_consensus.cc:729] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f324248b154fbe954996cf05f74711, State: Initialized, Role: FOLLOWER
I20260926 19:27:44.775090 8687 consensus_queue.cc:260] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.775923 8688 ts_tablet_manager.cc:1428] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:44.776260 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.775800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:54145)
0926 19:27:44.775955 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.776056 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 19:27:44.777413 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.776917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:33653)
0926 19:27:44.777151 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.777230 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 19:27:44.777851 8291 heartbeater.cc:503] Master 127.1.39.61:41101 was elected leader, sending a full tablet report...
I20260926 19:27:44.783152 8687 ts_tablet_manager.cc:1428] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260926 19:27:44.783771 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.783346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:37729)
0926 19:27:44.783525 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.783620 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 19:27:44.785385 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.784879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:35021)
0926 19:27:44.785064 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.785161 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 19:27:44.785611 8438 heartbeater.cc:503] Master 127.1.39.61:41101 was elected leader, sending a full tablet report...
W20260926 19:27:44.888687 8439 tablet.cc:2369] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:44.963349 8691 raft_consensus.cc:482] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:44.963667 8691 raft_consensus.cc:504] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.965262 8691 leader_election.cc:288] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309), f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:44.971752 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.965580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:33157 (local address 127.1.39.1:36369)
0926 19:27:44.965747 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.965768 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:44.965921 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:44.966332 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:44.966335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:44.966361 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:44.966705 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.966718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.967914 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.967920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.969449 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.969461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.970958 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.970966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.971150 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.971187 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:27:44.971289 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 19:27:44.971463 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 19:27:44.971835 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.965703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:36369 (local address 127.1.39.2:33157)
0926 19:27:44.965888 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:44.965894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:44.965919 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:27:44.966097 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:44.966105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:44.966192 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:44.966309 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:44.966879 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.967783 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.969621 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.970778 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.970892 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.970974 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:44.971348 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 19:27:44.971553 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:27:44.972370 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.965703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:34653 (local address 127.1.39.3:33309)
0926 19:27:44.965888 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:44.965894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:44.966468 (+ 574us) server_negotiation.cc:407] Connection header received
0926 19:27:44.966644 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:44.966651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:44.966729 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:44.966826 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:44.967351 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.968148 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.970350 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.971348 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.971457 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.971552 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:44.972000 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 19:27:44.972147 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:27:44.972558 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "827c49b220cf4c4a98ae0a3c38f06420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f324248b154fbe954996cf05f74711" is_pre_election: true
I20260926 19:27:44.972869 8391 raft_consensus.cc:2454] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827c49b220cf4c4a98ae0a3c38f06420 in term 0.
I20260926 19:27:44.973143 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.965580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:33309 (local address 127.1.39.1:34653)
0926 19:27:44.966354 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.966374 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:44.966512 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:44.966864 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:44.966868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:44.966890 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:44.967205 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.967216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.968306 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.968311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.970193 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:44.970208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:44.971562 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:44.971569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:44.971771 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:44.971804 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:27:44.972564 (+ 760us) client_negotiation.cc:240] Negotiation successful
0926 19:27:44.972705 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":139,"thread_start_us":312,"threads_started":1}
I20260926 19:27:44.973403 8179 leader_election.cc:302] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827c49b220cf4c4a98ae0a3c38f06420, f8f324248b154fbe954996cf05f74711; no voters:
I20260926 19:27:44.973889 8691 raft_consensus.cc:2781] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:44.974058 8691 raft_consensus.cc:482] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:44.974187 8691 raft_consensus.cc:3037] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:44.974201 8532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "827c49b220cf4c4a98ae0a3c38f06420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" is_pre_election: true
I20260926 19:27:44.974723 8532 raft_consensus.cc:2454] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 827c49b220cf4c4a98ae0a3c38f06420 in term 0.
I20260926 19:27:44.977221 8691 raft_consensus.cc:504] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.978041 8691 leader_election.cc:288] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [CANDIDATE]: Term 1 election: Requested vote from peers 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309), f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157)
I20260926 19:27:44.978396 8532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "827c49b220cf4c4a98ae0a3c38f06420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1684fc580c4f68b0230019e1f0e6bb"
I20260926 19:27:44.978627 8532 raft_consensus.cc:3037] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:44.978639 8391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "217eb2b58c73404d9e820298fc097659" candidate_uuid: "827c49b220cf4c4a98ae0a3c38f06420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f324248b154fbe954996cf05f74711"
I20260926 19:27:44.978940 8391 raft_consensus.cc:3037] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:44.981412 8532 raft_consensus.cc:2454] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827c49b220cf4c4a98ae0a3c38f06420 in term 1.
I20260926 19:27:44.981863 8391 raft_consensus.cc:2454] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 827c49b220cf4c4a98ae0a3c38f06420 in term 1.
I20260926 19:27:44.981948 8178 leader_election.cc:302] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 827c49b220cf4c4a98ae0a3c38f06420, 8c1684fc580c4f68b0230019e1f0e6bb; no voters:
I20260926 19:27:44.982301 8691 raft_consensus.cc:2781] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:44.982806 8691 raft_consensus.cc:686] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 LEADER]: Becoming Leader. State: Replica: 827c49b220cf4c4a98ae0a3c38f06420, State: Running, Role: LEADER
I20260926 19:27:44.983285 8691 consensus_queue.cc:237] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } }
I20260926 19:27:44.984325 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:44.983740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:36777)
0926 19:27:44.984034 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:44.984144 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260926 19:27:44.987273 7921 catalog_manager.cc:5495] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "827c49b220cf4c4a98ae0a3c38f06420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "827c49b220cf4c4a98ae0a3c38f06420" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33623 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:45.050658 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.044991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.1.39.1:33623)
0926 19:27:45.045195 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:45.045203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:45.045223 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:27:45.045361 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:45.045369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:45.045459 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:45.045583 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:45.045965 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.046809 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.048490 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.048880 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.048984 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:45.049469 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:45.050015 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:45.050139 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:45.050292 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:27:45.050356 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 19:27:45.054625 8391 raft_consensus.cc:1261] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Refusing update from remote peer 827c49b220cf4c4a98ae0a3c38f06420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:45.054991 8532 raft_consensus.cc:1261] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Refusing update from remote peer 827c49b220cf4c4a98ae0a3c38f06420: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:45.055390 8694 consensus_queue.cc:1035] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f324248b154fbe954996cf05f74711" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 33157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:45.055892 8691 consensus_queue.cc:1035] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1684fc580c4f68b0230019e1f0e6bb" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:45.058614 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.058196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:48845)
0926 19:27:45.058340 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.058427 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 19:27:45.058959 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.058407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:54407)
0926 19:27:45.058635 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.058753 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 19:27:45.066233 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.065762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.3:57819)
0926 19:27:45.065944 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.066031 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 19:27:45.066486 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.065787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.2:37003)
0926 19:27:45.066164 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.066271 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260926 19:27:45.070364 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.069632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:43567)
0926 19:27:45.070073 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.070197 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333}
I20260926 19:27:45.073173 8693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.072744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.1.39.1:48847)
0926 19:27:45.072884 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.072992 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 19:27:45.080094 7921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:45.080394 7921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:45.086457 7921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49931b8ce6234c9fbaf8dcd18c3aeb99: Sending DeleteTablet for 3 replicas of tablet 217eb2b58c73404d9e820298fc097659
I20260926 19:27:45.087445 8512 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb2b58c73404d9e820298fc097659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:45 UTC) from {username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:52174
I20260926 19:27:45.087555 8371 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb2b58c73404d9e820298fc097659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:45 UTC) from {username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:59106
I20260926 19:27:45.088279 8224 tablet_service.cc:1514] Processing DeleteTablet for tablet 217eb2b58c73404d9e820298fc097659 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:45 UTC) from {username='oryx', principal='oryx/127.1.39.61@KRBTEST.COM'} at 127.0.0.1:50306
I20260926 19:27:45.091602 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8124
I20260926 19:27:45.105502 8708 tablet_replica.cc:330] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: stopping tablet replica
I20260926 19:27:45.105731 8709 tablet_replica.cc:330] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: stopping tablet replica
I20260926 19:27:45.105921 8708 raft_consensus.cc:2229] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:45.105923 8709 raft_consensus.cc:2229] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:45.106328 8708 raft_consensus.cc:2258] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:45.106614 8709 raft_consensus.cc:2258] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:45.107218 8707 tablet_replica.cc:330] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: stopping tablet replica
I20260926 19:27:45.107424 8707 raft_consensus.cc:2229] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:45.107729 8707 raft_consensus.cc:2258] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:45.108174 8708 ts_tablet_manager.cc:1905] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:45.109282 8709 ts_tablet_manager.cc:1905] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:45.113888 8707 ts_tablet_manager.cc:1905] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:45.116756 8709 ts_tablet_manager.cc:1918] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:45.116945 8709 log.cc:1196] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/217eb2b58c73404d9e820298fc097659
I20260926 19:27:45.117486 8709 ts_tablet_manager.cc:1939] T 217eb2b58c73404d9e820298fc097659 P 827c49b220cf4c4a98ae0a3c38f06420: Deleting consensus metadata
I20260926 19:27:45.118908 7907 catalog_manager.cc:4841] TS 827c49b220cf4c4a98ae0a3c38f06420 (127.1.39.1:33623): tablet 217eb2b58c73404d9e820298fc097659 (table test-table [id=23b61aaa92394089b9513b6493777c14]) successfully deleted
I20260926 19:27:45.120609 8708 ts_tablet_manager.cc:1918] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:45.120675 8707 ts_tablet_manager.cc:1918] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:45.120808 8708 log.cc:1196] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/217eb2b58c73404d9e820298fc097659
I20260926 19:27:45.120828 8707 log.cc:1196] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/217eb2b58c73404d9e820298fc097659
I20260926 19:27:45.121315 8708 ts_tablet_manager.cc:1939] T 217eb2b58c73404d9e820298fc097659 P f8f324248b154fbe954996cf05f74711: Deleting consensus metadata
I20260926 19:27:45.121331 8707 ts_tablet_manager.cc:1939] T 217eb2b58c73404d9e820298fc097659 P 8c1684fc580c4f68b0230019e1f0e6bb: Deleting consensus metadata
I20260926 19:27:45.122522 7907 catalog_manager.cc:4841] TS 8c1684fc580c4f68b0230019e1f0e6bb (127.1.39.3:33309): tablet 217eb2b58c73404d9e820298fc097659 (table test-table [id=23b61aaa92394089b9513b6493777c14]) successfully deleted
I20260926 19:27:45.122540 7908 catalog_manager.cc:4841] TS f8f324248b154fbe954996cf05f74711 (127.1.39.2:33157): tablet 217eb2b58c73404d9e820298fc097659 (table test-table [id=23b61aaa92394089b9513b6493777c14]) successfully deleted
I20260926 19:27:45.124468 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.093525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.1.39.1:33623)
0926 19:27:45.093708 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:45.093714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:45.104641 (+ 10927us) server_negotiation.cc:407] Connection header received
0926 19:27:45.104886 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:45.104891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:45.104983 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:45.105137 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:45.107190 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.108479 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.110388 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.111074 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.111177 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:45.114962 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:45.114977 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:45.115000 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:45.115047 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:45.120085 (+ 5038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.121228 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.121239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.121243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.121486 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.122009 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.122015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.122018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.123161 (+ 1143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:45.123494 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:45.123842 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 19:27:45.124080 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 19:27:45.125190 8284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:45.238436 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8124
I20260926 19:27:45.258615 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8301
I20260926 19:27:45.268970 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.268137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56460)
0926 19:27:45.268628 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.268738 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":222,"threads_started":1}
I20260926 19:27:45.270357 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.269921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:33725 (local address 127.0.0.1:56472)
0926 19:27:45.270102 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:45.270206 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:33725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:27:45.273613 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.259408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.1.39.2:33157)
0926 19:27:45.259611 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:45.259619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:45.259640 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:45.259850 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:45.259855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:45.259915 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:45.260016 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:45.261281 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.262080 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.263289 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.263709 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.263885 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:45.266433 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:45.266457 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:45.266474 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:45.266527 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:45.270373 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.271165 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.271176 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.271182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.271326 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.271710 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.271717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.271720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.272111 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:45.272434 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:45.272720 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:27:45.272950 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:27:45.274286 8431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:45.369936 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8301
I20260926 19:27:45.387794 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8449
I20260926 19:27:45.400360 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:45.388539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52184 (local address 127.1.39.3:33309)
0926 19:27:45.388711 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:45.388715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:45.388734 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:45.388861 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:45.388864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:45.388928 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:45.389010 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:45.390405 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.391085 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.392229 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:45.392597 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:45.392684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:45.395231 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:45.395246 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:45.395263 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:45.395300 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:45.397960 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.398558 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.398565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.398567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.398684 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:45.398994 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:45.398997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:45.398999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:45.399363 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:45.399567 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:45.399836 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 19:27:45.400000 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:27:45.401048 8572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:45.518584 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8449
I20260926 19:27:45.541497 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7783
I20260926 19:27:45.542670 7952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:45.679149 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7783
I20260926 19:27:45.702852 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 7884
I20260926 19:27:45.704072 8141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:45.818929 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 7884
2026-09-26T19:27:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11130 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:45 dist-test-slave-9wb0 krb5kdc[8720](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:45 dist-test-slave-9wb0 krb5kdc[8720](info): set up 2 sockets
Sep 26 19:27:45 dist-test-slave-9wb0 krb5kdc[8720](info): commencing operation
krb5kdc: starting...
W20260926 19:27:46.939221 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:47 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab.
Entry for principal oryx/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab.
I20260926 19:27:47.168844 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37115
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab
--principal=oryx/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:37115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:47.493554 8733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:47.493839 8733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:47.494041 8733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:47.501171 8733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:47.501273 8733 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 19:27:47.501374 8733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:47.501471 8733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:47.501571 8733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:47.501675 8733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:47.511350 8733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:37115
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.62.keytab
--principal=oryx/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:47.512686 8733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:47.514071 8733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:47.526198 8741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:47.526196 8739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:47.526214 8738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:47.526536 8733 server_base.cc:1034] running on GCE node
I20260926 19:27:47.527563 8733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:47.528936 8733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:47.530162 8733 hybrid_clock.cc:648] HybridClock initialized: now 1790450867530079 us; error 46 us; skew 500 ppm
I20260926 19:27:47.530519 8733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:47 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:47.541226 8733 init.cc:377] Logged in from keytab as oryx/127.1.39.62@KRBTEST.COM (short username oryx)
I20260926 19:27:47.542814 8733 webserver.cc:460] Webserver started at http://127.1.39.62:35269/ using document root <none> and password file <none>
I20260926 19:27:47.543537 8733 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:47.543628 8733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:47.543895 8733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:47.546905 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "214cf3628c604becb061f1e7c65395b3"
format_stamp: "Formatted at 2026-09-26 19:27:47 on dist-test-slave-9wb0"
server_key: "8a72f30e2457c323c49998a52aad9efc"
server_key_iv: "7366f9f728a205f1ec885c09da04ca48"
server_key_version: "encryptionkey@0"
I20260926 19:27:47.547699 8733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "214cf3628c604becb061f1e7c65395b3"
format_stamp: "Formatted at 2026-09-26 19:27:47 on dist-test-slave-9wb0"
server_key: "8a72f30e2457c323c49998a52aad9efc"
server_key_iv: "7366f9f728a205f1ec885c09da04ca48"
server_key_version: "encryptionkey@0"
I20260926 19:27:47.553095 8733 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:27:47.557134 8748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:47.558729 8733 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:47.558947 8733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "214cf3628c604becb061f1e7c65395b3"
format_stamp: "Formatted at 2026-09-26 19:27:47 on dist-test-slave-9wb0"
server_key: "8a72f30e2457c323c49998a52aad9efc"
server_key_iv: "7366f9f728a205f1ec885c09da04ca48"
server_key_version: "encryptionkey@0"
I20260926 19:27:47.559203 8733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:47.584548 8733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:47.591730 8733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:47.592042 8733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:47.617624 8733 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:37115
I20260926 19:27:47.617647 8809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:37115 every 8 connection(s)
I20260926 19:27:47.619552 8733 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:47.620772 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8733
I20260926 19:27:47.620977 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:47.621803 1180 external_mini_cluster.cc:1428] Setting key a058d9240e7de909eeb3b28f0087b4d6
I20260926 19:27:47.624504 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:47.637928 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Bootstrap starting.
Sep 26 19:27:47 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:47.641239 8810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:47.642411 8810 log.cc:824] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:47.645063 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: No bootstrap required, opened a new log
I20260926 19:27:47.646931 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:47.623877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.1.39.62:37115)
0926 19:27:47.624738 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:47.624756 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:47.624818 (+ 62us) server_negotiation.cc:407] Connection header received
0926 19:27:47.627598 (+ 2780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:47.627629 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:47.628199 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:47.628680 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:47.630409 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:47.632729 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:47.634620 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:47.635322 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:47.635463 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:47.639943 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:47.639950 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:47.639964 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:47.639997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:47.643158 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:47.643837 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:47.643843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:47.643849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:47.644003 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:47.644370 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:47.644377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:47.644385 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:47.644874 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:47.645182 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:47.645616 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 19:27:47.646002 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":365,"threads_started":1}
I20260926 19:27:47.653565 8810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } }
I20260926 19:27:47.654012 8810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:47.654116 8810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cf3628c604becb061f1e7c65395b3, State: Initialized, Role: FOLLOWER
I20260926 19:27:47.654672 8810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } }
I20260926 19:27:47.654891 8810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:47.655040 8810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:47.655210 8810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:47.657531 8810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } }
I20260926 19:27:47.658058 8810 leader_election.cc:302] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214cf3628c604becb061f1e7c65395b3; no voters:
I20260926 19:27:47.658607 8810 leader_election.cc:288] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:47.658742 8815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:47.659485 8815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [term 1 LEADER]: Becoming Leader. State: Replica: 214cf3628c604becb061f1e7c65395b3, State: Running, Role: LEADER
I20260926 19:27:47.659972 8815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } }
I20260926 19:27:47.660679 8810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:47.662979 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214cf3628c604becb061f1e7c65395b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } } }
I20260926 19:27:47.663365 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:47.662943 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214cf3628c604becb061f1e7c65395b3. Latest consensus state: current_term: 1 leader_uuid: "214cf3628c604becb061f1e7c65395b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214cf3628c604becb061f1e7c65395b3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37115 } } }
I20260926 19:27:47.664872 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:47.666320 8820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:47.671993 8820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:47.681200 8820 catalog_manager.cc:1347] Generated new cluster ID: d53f728e5e174b10b41b585945730eb1
I20260926 19:27:47.681350 8820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:47.711509 8820 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:47.713626 8820 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:47.730933 8820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Generated new TSK 0
I20260926 19:27:47.731806 8820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
I20260926 19:27:47.802238 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37115
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:48.147243 8836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:48.147511 8836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:48.147604 8836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:48.154971 8836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:48.155081 8836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:48.155169 8836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:48.155247 8836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:48.165431 8836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:37115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:48.166894 8836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:48.168308 8836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:48.181862 8844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:48.181967 8841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:48.181933 8842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:48.182557 8836 server_base.cc:1034] running on GCE node
I20260926 19:27:48.183193 8836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:48.184093 8836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:48.185382 8836 hybrid_clock.cc:648] HybridClock initialized: now 1790450868185255 us; error 61 us; skew 500 ppm
I20260926 19:27:48.185678 8836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:48 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:48.188867 8836 init.cc:377] Logged in from keytab as oryx/127.1.39.1@KRBTEST.COM (short username oryx)
I20260926 19:27:48.190337 8836 webserver.cc:460] Webserver started at http://127.1.39.1:42401/ using document root <none> and password file <none>
I20260926 19:27:48.191025 8836 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:48.191157 8836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:48.191483 8836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:48.194454 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "63adcabb216a92509e1d91e71fd946fc"
server_key_iv: "9f539ce9d5b1dc8356d73037579e77ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.195426 8836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "63adcabb216a92509e1d91e71fd946fc"
server_key_iv: "9f539ce9d5b1dc8356d73037579e77ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.200560 8836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:27:48.203904 8851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:48.205222 8836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:48.205492 8836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "63adcabb216a92509e1d91e71fd946fc"
server_key_iv: "9f539ce9d5b1dc8356d73037579e77ac"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.205735 8836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:48.232328 8836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:48.236691 8836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:48.236984 8836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:48.239616 8836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:48.239727 8836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:48.239881 8836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:48.239961 8836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:48 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:48.264478 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.240398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:40699 (local address 127.1.39.62:37115)
0926 19:27:48.240843 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:48.240849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:48.242065 (+ 1216us) server_negotiation.cc:407] Connection header received
0926 19:27:48.244061 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:48.244066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:48.244149 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:48.244265 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:48.247080 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.248009 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.251217 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.251711 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.251880 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.257472 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:48.257478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:48.257481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:48.257513 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:48.260074 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.261241 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.261246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.261251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.261407 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.262001 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.262009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.262015 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.262473 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:48.262652 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:48.263605 (+ 953us) server_negotiation.cc:299] Negotiation successful
0926 19:27:48.263927 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260926 19:27:48.265101 8861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.241018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.1:40699)
0926 19:27:48.241883 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:48.241926 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:48.243789 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:48.244726 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:48.244742 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:48.245715 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:48.246857 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.246896 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.248214 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.248220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.251048 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.251069 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.252004 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.252014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.252380 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.253210 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:48.253248 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:48.257171 (+ 3923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:48.260217 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.260230 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.260253 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.261113 (+ 860us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.261530 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.261536 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.261540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.261832 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.262625 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:48.262632 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:48.263107 (+ 475us) client_negotiation.cc:769] Sending connection context
0926 19:27:48.263327 (+ 220us) client_negotiation.cc:240] Negotiation successful
0926 19:27:48.263794 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":397,"threads_started":1}
I20260926 19:27:48.300194 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.277346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:55127 (local address 127.1.39.62:37115)
0926 19:27:48.277569 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:48.277578 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:48.282018 (+ 4440us) server_negotiation.cc:407] Connection header received
0926 19:27:48.282174 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:48.282182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:48.282289 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:48.282414 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:48.284332 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.285693 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.288690 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.289520 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.289688 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.292515 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:48.292525 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:48.292533 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:48.292584 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:48.296528 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.297682 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.297689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.297693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.297853 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.298422 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.298429 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.298436 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.298829 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:48.298960 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:48.299455 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 19:27:48.299721 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:27:48.300330 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.277764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.1:55127)
0926 19:27:48.281859 (+ 4095us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:48.281892 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:48.282048 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:48.282499 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:48.282507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:48.283437 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:48.284109 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.284132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.285861 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.285874 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.288491 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.288512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.289783 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.289798 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.290057 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.291048 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:48.291092 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:48.292240 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:48.296690 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.296700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.296705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.297525 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.298025 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.298031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.298034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.298221 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.298979 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:48.298983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:48.299170 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 19:27:48.299311 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:27:48.299779 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3899,"thread_start_us":3710,"threads_started":1}
I20260926 19:27:48.302140 8836 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33919
I20260926 19:27:48.302337 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33919 every 8 connection(s)
I20260926 19:27:48.303275 8836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:48.306097 8972 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37115
I20260926 19:27:48.306361 8972 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:48.307080 8972 heartbeater.cc:511] Master 127.1.39.62:37115 requested a full tablet report, sending...
I20260926 19:27:48.309216 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8836
I20260926 19:27:48.309499 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:48.309571 8765 ts_manager.cc:194] Registered new tserver with Master: 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919)
I20260926 19:27:48.310526 1180 external_mini_cluster.cc:1428] Setting key 4987e0910b40b87ab437bbcd35f36cd6
I20260926 19:27:48.312520 8765 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:40699
WARNING: no policy specified for oryx/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
I20260926 19:27:48.360870 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37115
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:48.688592 8976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:48.688967 8976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:48.689190 8976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:48.696218 8976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:48.696441 8976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:48.696693 8976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:48.696848 8976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:48.706305 8976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:37115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:48.707554 8976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:48.708820 8976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:48.721535 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:48.721495 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:48.721580 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:48.722352 8976 server_base.cc:1034] running on GCE node
I20260926 19:27:48.722918 8976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:48.723776 8976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:48.725014 8976 hybrid_clock.cc:648] HybridClock initialized: now 1790450868724925 us; error 59 us; skew 500 ppm
I20260926 19:27:48.725286 8976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:48 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:48.728255 8976 init.cc:377] Logged in from keytab as oryx/127.1.39.2@KRBTEST.COM (short username oryx)
I20260926 19:27:48.729689 8976 webserver.cc:460] Webserver started at http://127.1.39.2:37273/ using document root <none> and password file <none>
I20260926 19:27:48.730361 8976 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:48.730491 8976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:48.730799 8976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:48.733696 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "944eb67c51a042a08dfe1d89c64ba304"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "de9d770d985736814bd682d467d6b6c7"
server_key_iv: "6c2ceb0fccda268f0f372b6553dd4c90"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.734449 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "944eb67c51a042a08dfe1d89c64ba304"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "de9d770d985736814bd682d467d6b6c7"
server_key_iv: "6c2ceb0fccda268f0f372b6553dd4c90"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.739223 8976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:27:48.742523 8991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:48.743818 8976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:48.743988 8976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "944eb67c51a042a08dfe1d89c64ba304"
format_stamp: "Formatted at 2026-09-26 19:27:48 on dist-test-slave-9wb0"
server_key: "de9d770d985736814bd682d467d6b6c7"
server_key_iv: "6c2ceb0fccda268f0f372b6553dd4c90"
server_key_version: "encryptionkey@0"
I20260926 19:27:48.744210 8976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:48.759202 8976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:48.763684 8976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:48.763967 8976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:48.767175 8976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:48.767284 8976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:48.767428 8976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:48.767503 8976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:48 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450868, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:48.790038 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.767219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:49327 (local address 127.1.39.62:37115)
0926 19:27:48.767408 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:48.767414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:48.769020 (+ 1606us) server_negotiation.cc:407] Connection header received
0926 19:27:48.771046 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:48.771051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:48.771132 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:48.771258 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:48.773129 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.774169 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.776811 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.777195 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.777370 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.782600 (+ 5230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:48.782605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:48.782609 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:48.782637 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:48.785876 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.786871 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.786878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.786883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.787049 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.787637 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.787646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.787653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.788132 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:48.788324 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:48.789490 (+ 1166us) server_negotiation.cc:299] Negotiation successful
0926 19:27:48.789689 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:27:48.791267 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.767963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.2:49327)
0926 19:27:48.768762 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:48.768892 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:48.770777 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:48.771519 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:48.771533 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:48.772195 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:48.772968 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.772987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.774333 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.774339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.776609 (+ 2270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.776629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.777437 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.777448 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.777694 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.778561 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:48.778584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:48.782306 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:48.786020 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.786033 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.786056 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.786733 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.787197 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.787206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.787210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.787478 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.788273 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:48.788282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:48.788681 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 19:27:48.788987 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 19:27:48.789540 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":338,"threads_started":1}
I20260926 19:27:48.814251 8976 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:37969
I20260926 19:27:48.814276 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:37969 every 8 connection(s)
I20260926 19:27:48.815130 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.799434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.2:51921)
0926 19:27:48.801952 (+ 2518us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:48.801972 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:48.802205 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:48.802577 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:48.802582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:48.803014 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:48.803288 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.803296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.804540 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.804549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.806115 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:48.806126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.807203 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.807208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:48.807328 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.808158 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:48.808186 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:48.808858 (+ 672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:48.812109 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.812119 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.812126 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.812709 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.813120 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:48.813126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:48.813132 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:48.813365 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:48.814236 (+ 871us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:48.814244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:48.814466 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 19:27:48.814607 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 19:27:48.814830 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2253,"thread_start_us":2112,"threads_started":1}
I20260926 19:27:48.815341 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:48.799895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:51921 (local address 127.1.39.62:37115)
0926 19:27:48.800040 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:48.800044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:48.802078 (+ 2034us) server_negotiation.cc:407] Connection header received
0926 19:27:48.802366 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:48.802373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:48.802445 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:48.802555 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:48.803419 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.804406 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.806256 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:48.806794 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:48.806946 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:48.809049 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:48.809057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:48.809064 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:48.809109 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:48.811966 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.812827 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.812831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.812833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.812938 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:48.813577 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:48.813584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:48.813592 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:48.814100 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:48.814266 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:48.814767 (+ 501us) server_negotiation.cc:299] Negotiation successful
0926 19:27:48.814997 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 19:27:48.815824 8976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:48.818253 9111 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37115
I20260926 19:27:48.818480 9111 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:48.819114 9111 heartbeater.cc:511] Master 127.1.39.62:37115 requested a full tablet report, sending...
I20260926 19:27:48.820322 8765 ts_manager.cc:194] Registered new tserver with Master: 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969)
I20260926 19:27:48.821041 8765 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:49327
I20260926 19:27:48.822376 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 8976
I20260926 19:27:48.822564 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:48.822973 1180 external_mini_cluster.cc:1428] Setting key f4b75d27b27d1cab61fca8fe4dfc9ced
WARNING: no policy specified for oryx/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
I20260926 19:27:48.870334 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37115
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:49.184904 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:49.185210 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:49.185406 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:49.192579 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:49.192723 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:49.192876 9115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 19:27:49.193017 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:49.202764 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.39.62:37115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:49.204016 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:49.205391 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:49.218252 9120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:49.218303 9121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:49.218216 9123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:49.219381 9115 server_base.cc:1034] running on GCE node
I20260926 19:27:49.219972 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:49.220821 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:49.222064 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790450869222001 us; error 36 us; skew 500 ppm
I20260926 19:27:49.222359 9115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:49.225481 9115 init.cc:377] Logged in from keytab as oryx/127.1.39.3@KRBTEST.COM (short username oryx)
I20260926 19:27:49.226845 9115 webserver.cc:460] Webserver started at http://127.1.39.3:35337/ using document root <none> and password file <none>
I20260926 19:27:49.227504 9115 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:49.227589 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:49.227902 9115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:49.230919 9115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "14c4c76aaf3a41bcaef3e0700e451502"
format_stamp: "Formatted at 2026-09-26 19:27:49 on dist-test-slave-9wb0"
server_key: "836b3646a090a53dc3fee6c7adb18441"
server_key_iv: "99d8408ee21f629dff366e210fbcfd6b"
server_key_version: "encryptionkey@0"
I20260926 19:27:49.231709 9115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "14c4c76aaf3a41bcaef3e0700e451502"
format_stamp: "Formatted at 2026-09-26 19:27:49 on dist-test-slave-9wb0"
server_key: "836b3646a090a53dc3fee6c7adb18441"
server_key_iv: "99d8408ee21f629dff366e210fbcfd6b"
server_key_version: "encryptionkey@0"
I20260926 19:27:49.237072 9115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:27:49.240795 9130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:49.242026 9115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:49.242188 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "14c4c76aaf3a41bcaef3e0700e451502"
format_stamp: "Formatted at 2026-09-26 19:27:49 on dist-test-slave-9wb0"
server_key: "836b3646a090a53dc3fee6c7adb18441"
server_key_iv: "99d8408ee21f629dff366e210fbcfd6b"
server_key_version: "encryptionkey@0"
I20260926 19:27:49.242389 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:49.283219 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:49.287252 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:49.287539 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:49.290232 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:49.290341 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:49.290503 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:49.290607 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450869, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:49.315241 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.290842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:47329 (local address 127.1.39.62:37115)
0926 19:27:49.291062 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.291068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.292626 (+ 1558us) server_negotiation.cc:407] Connection header received
0926 19:27:49.294905 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.294911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.294997 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.295113 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:49.297347 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.298689 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.301332 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.301962 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.302076 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.307182 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:49.307191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:49.307198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:49.307245 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:49.310809 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.312007 (+ 1198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.312012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.312015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.312198 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.312706 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.312713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.312717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.313137 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:49.313279 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.314417 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.314730 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 19:27:49.315616 8972 heartbeater.cc:503] Master 127.1.39.62:37115 was elected leader, sending a full tablet report...
I20260926 19:27:49.316666 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.291487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.3:47329)
0926 19:27:49.292446 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.292499 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.294612 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.295371 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.295385 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.296232 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:49.297119 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.297152 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.298858 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.298868 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.301121 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.301144 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.302372 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.302380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.302648 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.303458 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:49.303508 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:49.306932 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:49.310964 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.310978 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.311002 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.311814 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.312350 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.312360 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.312364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.312575 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.313385 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:49.313392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:49.313824 (+ 432us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.314138 (+ 314us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.314665 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":492,"threads_started":1}
I20260926 19:27:49.336724 9201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.323241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.1.39.3:43591)
0926 19:27:49.324309 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.324325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.324426 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.324784 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.324787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.325202 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:49.325580 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.325589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.326525 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.326532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.327520 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.327531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.328249 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.328255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.328418 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.329024 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:49.329040 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:49.330017 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:49.333763 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.333772 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.333776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.334369 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.334741 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.334746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.334750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.334922 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.335645 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:49.335650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:49.335857 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.336041 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.336394 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":788,"threads_started":1}
I20260926 19:27:49.337493 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.323208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:43591 (local address 127.1.39.62:37115)
0926 19:27:49.323427 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.323436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.324401 (+ 965us) server_negotiation.cc:407] Connection header received
0926 19:27:49.324529 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.324537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.324646 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.324775 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:49.325719 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.326395 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.327659 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.328074 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.328209 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.330224 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:49.330230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:49.330235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:49.330268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:49.333636 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.334502 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.334507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.334511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.334631 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.335084 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.335092 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.335096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.335415 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:49.335570 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.337045 (+ 1475us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.337196 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:27:49.345947 9115 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:35651
I20260926 19:27:49.345985 9249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:35651 every 8 connection(s)
I20260926 19:27:49.347134 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:49.349087 9250 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37115
I20260926 19:27:49.349300 9250 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:49.349949 9250 heartbeater.cc:511] Master 127.1.39.62:37115 requested a full tablet report, sending...
I20260926 19:27:49.351194 8765 ts_manager.cc:194] Registered new tserver with Master: 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:49.351974 8765 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:47329
I20260926 19:27:49.352501 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9115
I20260926 19:27:49.352680 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:49.353102 1180 external_mini_cluster.cc:1428] Setting key a9411c6c8aba8f17e9d4cced879bae6b
I20260926 19:27:49.355935 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:27:49.405349 8860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:49.395262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.1.39.62:37115)
0926 19:27:49.395510 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.395518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.395800 (+ 282us) server_negotiation.cc:407] Connection header received
0926 19:27:49.395939 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.395944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.396024 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.396118 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:49.397670 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.398377 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.399550 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.399934 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.400024 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.405082 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56038: BlockingRecv error: recv got EOF from 127.0.0.1:56038 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450869, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:49.421418 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.408511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.1.39.62:37115)
0926 19:27:49.408736 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.408741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.409020 (+ 279us) server_negotiation.cc:407] Connection header received
0926 19:27:49.409198 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.409204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.409284 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.409419 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:49.410578 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.411271 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.412501 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.412889 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.413026 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.416517 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:49.416523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:49.416526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:49.416556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:49.419109 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.419835 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.419839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.419841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.419969 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.420290 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.420293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.420295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.420551 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:49.420757 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.420975 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.421125 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:27:49.425187 8765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:49.431864 8765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:49.453469 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.443567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48696 (local address 127.1.39.1:33919)
0926 19:27:49.445702 (+ 2135us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.445710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.446373 (+ 663us) server_negotiation.cc:407] Connection header received
0926 19:27:49.446473 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.446481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.446771 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.447678 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.447902 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.449171 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.451288 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.452359 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.452515 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.452724 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.452998 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.453150 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":1901,"threads_started":1}
I20260926 19:27:49.453658 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.443498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33919 (local address 127.0.0.1:48696)
0926 19:27:49.446163 (+ 2665us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.446180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.446303 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.446969 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.446977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.447006 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.447438 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.447449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.449332 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.449338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.451112 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.451127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.452544 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.452549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.452772 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.452816 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.452947 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.453135 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2524,"mutex_wait_us":68,"thread_start_us":2374,"threads_started":1}
I20260926 19:27:49.454104 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.443478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35651 (local address 127.0.0.1:58748)
0926 19:27:49.444262 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.444308 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.444478 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.446077 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.446084 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.446120 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.446608 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.446620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.448759 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.448766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.451381 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.451393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.453215 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.453221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.453406 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.453445 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.453511 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.453634 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":33,"thread_start_us":847,"threads_started":1}
I20260926 19:27:49.454501 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.444095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.1.39.3:35651)
0926 19:27:49.445481 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.445489 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.445533 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:27:49.445628 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.445635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.445889 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.446073 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.446844 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.448585 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.451573 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.452998 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.453743 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.453947 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.454024 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.454181 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1210,"thread_start_us":1020,"threads_started":1}
I20260926 19:27:49.455704 9273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.446759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:37969 (local address 127.0.0.1:41396)
0926 19:27:49.447674 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.447693 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.447862 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.448545 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.448548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.448565 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.449116 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.449127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.450937 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.450947 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.453069 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.453086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.454995 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.455000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.455104 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.455125 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.455224 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.455350 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":1,"thread_start_us":439,"threads_started":1}
I20260926 19:27:49.456346 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.446449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.1.39.2:37969)
0926 19:27:49.446994 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.447001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.447778 (+ 777us) server_negotiation.cc:407] Connection header received
0926 19:27:49.448033 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.448040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.448248 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.448372 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.449339 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.450774 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.453236 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.454567 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.454688 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.454865 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.455282 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.456063 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":406,"threads_started":1}
I20260926 19:27:49.457975 8902 tablet_service.cc:1467] Processing CreateTablet for tablet 24b23c7d32dc428089b89ef106618706 (DEFAULT_TABLE table=test-table [id=abe9f92ce26f4c05bea17c547a190c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.458065 9180 tablet_service.cc:1467] Processing CreateTablet for tablet 24b23c7d32dc428089b89ef106618706 (DEFAULT_TABLE table=test-table [id=abe9f92ce26f4c05bea17c547a190c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.459362 8902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b23c7d32dc428089b89ef106618706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.459411 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b23c7d32dc428089b89ef106618706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.459470 9041 tablet_service.cc:1467] Processing CreateTablet for tablet 24b23c7d32dc428089b89ef106618706 (DEFAULT_TABLE table=test-table [id=abe9f92ce26f4c05bea17c547a190c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.460704 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24b23c7d32dc428089b89ef106618706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.471912 9274 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Bootstrap starting.
I20260926 19:27:49.474059 9275 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Bootstrap starting.
I20260926 19:27:49.475628 9276 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Bootstrap starting.
I20260926 19:27:49.476851 9274 tablet_bootstrap.cc:654] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.477384 9275 tablet_bootstrap.cc:654] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.478197 9274 log.cc:824] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:49.478328 9275 log.cc:824] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:49.478920 9276 tablet_bootstrap.cc:654] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.480007 9276 log.cc:824] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:49.481230 9274 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: No bootstrap required, opened a new log
I20260926 19:27:49.481230 9275 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: No bootstrap required, opened a new log
I20260926 19:27:49.481519 9275 ts_tablet_manager.cc:1397] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260926 19:27:49.481505 9274 ts_tablet_manager.cc:1397] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 19:27:49.482369 9276 tablet_bootstrap.cc:492] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: No bootstrap required, opened a new log
I20260926 19:27:49.482622 9276 ts_tablet_manager.cc:1397] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 19:27:49.489660 9276 raft_consensus.cc:348] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.490010 9276 raft_consensus.cc:374] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.490131 9276 raft_consensus.cc:729] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.490724 9276 consensus_queue.cc:260] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.491720 9111 heartbeater.cc:503] Master 127.1.39.62:37115 was elected leader, sending a full tablet report...
I20260926 19:27:49.492019 9275 raft_consensus.cc:348] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.492022 9274 raft_consensus.cc:348] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.492410 9275 raft_consensus.cc:374] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.492410 9274 raft_consensus.cc:374] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.492509 9274 raft_consensus.cc:729] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc227cad664fc5a6ba653eebfa1ce0, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.492524 9275 raft_consensus.cc:729] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.493075 9275 consensus_queue.cc:260] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.493136 9274 consensus_queue.cc:260] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.493561 9276 ts_tablet_manager.cc:1428] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260926 19:27:49.495103 9275 ts_tablet_manager.cc:1428] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 19:27:49.495455 9250 heartbeater.cc:503] Master 127.1.39.62:37115 was elected leader, sending a full tablet report...
I20260926 19:27:49.496073 9274 ts_tablet_manager.cc:1428] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 19:27:49.515054 9280 raft_consensus.cc:482] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:49.515316 9280 raft_consensus.cc:504] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.516842 9280 leader_election.cc:288] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:49.525243 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.517666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:41323 (local address 127.1.39.1:33919)
0926 19:27:49.517883 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.517888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.518308 (+ 420us) server_negotiation.cc:407] Connection header received
0926 19:27:49.518531 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.518535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.518606 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.518761 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.519341 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.520454 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.523200 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.524252 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.524405 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.524507 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.524867 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.524998 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 19:27:49.525370 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.517406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:32799 (local address 127.1.39.3:35651)
0926 19:27:49.517615 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.517622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.518596 (+ 974us) server_negotiation.cc:407] Connection header received
0926 19:27:49.518823 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.518828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.518902 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.518981 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.519614 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.520600 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.523160 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.524265 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.524385 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.524506 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.524962 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.525099 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:27:49.525463 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.517284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35651 (local address 127.1.39.2:32799)
0926 19:27:49.518495 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.518514 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.518667 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.519048 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.519052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.519094 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.519463 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.519471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.520736 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.520742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.522981 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.522998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.524675 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.524680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.524801 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.524823 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.524926 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.525046 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":31,"thread_start_us":462,"threads_started":1}
I20260926 19:27:49.525784 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.517156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33919 (local address 127.1.39.2:41323)
0926 19:27:49.518162 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.518185 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.518355 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.518767 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.518774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.518804 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.519181 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.519198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.520584 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.520588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.522984 (+ 2396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.522998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.524465 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.524472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.524701 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.524736 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.524841 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.524967 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":3,"thread_start_us":982,"threads_started":1}
I20260926 19:27:49.526592 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b23c7d32dc428089b89ef106618706" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502" is_pre_election: true
I20260926 19:27:49.526624 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b23c7d32dc428089b89ef106618706" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" is_pre_election: true
I20260926 19:27:49.527063 8927 raft_consensus.cc:2454] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:49.527068 9205 raft_consensus.cc:2454] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:49.527690 8992 leader_election.cc:302] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.528075 9280 raft_consensus.cc:2781] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:49.528218 9280 raft_consensus.cc:482] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:49.528339 9280 raft_consensus.cc:3037] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.530503 9280 raft_consensus.cc:504] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.531091 9280 leader_election.cc:288] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:49.531517 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b23c7d32dc428089b89ef106618706" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
I20260926 19:27:49.531545 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24b23c7d32dc428089b89ef106618706" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502"
I20260926 19:27:49.531719 8927 raft_consensus.cc:3037] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.531719 9205 raft_consensus.cc:3037] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.534839 9205 raft_consensus.cc:2454] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:49.534838 8927 raft_consensus.cc:2454] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:49.535236 8992 leader_election.cc:302] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.535485 9280 raft_consensus.cc:2781] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:49.535979 9280 raft_consensus.cc:686] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Becoming Leader. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Running, Role: LEADER
I20260926 19:27:49.536422 9280 consensus_queue.cc:237] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } }
I20260926 19:27:49.540508 8765 catalog_manager.cc:5495] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2). New cstate: current_term: 1 leader_uuid: "944eb67c51a042a08dfe1d89c64ba304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:49.554778 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.549928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.1.39.2:37969)
0926 19:27:49.550113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.550118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.550133 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:27:49.550303 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.550307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.550371 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.550481 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:49.550769 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.551454 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.552723 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.553123 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.553210 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.553671 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:49.554162 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:49.554262 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.554432 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.554493 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 19:27:49.556188 8973 tablet.cc:2369] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:49.562496 8927 raft_consensus.cc:1261] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.562494 9205 raft_consensus.cc:1261] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.563441 9285 consensus_queue.cc:1035] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:49.563782 9280 consensus_queue.cc:1035] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 19:27:49.568011 9112 tablet.cc:2369] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:49.579638 9288 mvcc.cc:205] Tried to move back new op lower bound from 7333686761718542336 to 7333686761624940544. Current Snapshot: MvccSnapshot[applied={T|T < 7333686761718542336}]
I20260926 19:27:49.583727 9289 mvcc.cc:205] Tried to move back new op lower bound from 7333686761718542336 to 7333686761624940544. Current Snapshot: MvccSnapshot[applied={T|T < 7333686761718542336}]
I20260926 19:27:49.598958 8765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:49.599277 8765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:49.602941 8765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Sending DeleteTablet for 3 replicas of tablet 24b23c7d32dc428089b89ef106618706
I20260926 19:27:49.604080 8902 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b23c7d32dc428089b89ef106618706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:49 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48696
I20260926 19:27:49.604355 9180 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b23c7d32dc428089b89ef106618706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:49 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260926 19:27:49.605647 9041 tablet_service.cc:1514] Processing DeleteTablet for tablet 24b23c7d32dc428089b89ef106618706 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:49 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:41396
I20260926 19:27:49.605517 8765 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:49.606513 8765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:49.608914 9300 tablet_replica.cc:330] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: stopping tablet replica
I20260926 19:27:49.609561 9300 raft_consensus.cc:2229] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:49.610178 9300 raft_consensus.cc:2258] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:49.615163 9300 ts_tablet_manager.cc:1905] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:49.616228 8901 tablet_service.cc:1467] Processing CreateTablet for tablet 887185d319d341e28ebdb7145bd6faf8 (DEFAULT_TABLE table=test-table [id=077b43bcdb8d422b915948351db9e919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.616955 8901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887185d319d341e28ebdb7145bd6faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.622681 9301 tablet_replica.cc:330] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: stopping tablet replica
I20260926 19:27:49.623104 9301 raft_consensus.cc:2229] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:49.623891 9301 raft_consensus.cc:2258] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:49.626406 9040 tablet_service.cc:1467] Processing CreateTablet for tablet 887185d319d341e28ebdb7145bd6faf8 (DEFAULT_TABLE table=test-table [id=077b43bcdb8d422b915948351db9e919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.626729 9300 ts_tablet_manager.cc:1918] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:49.626987 9300 log.cc:1196] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/24b23c7d32dc428089b89ef106618706
I20260926 19:27:49.627020 9040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887185d319d341e28ebdb7145bd6faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.627640 9300 ts_tablet_manager.cc:1939] T 24b23c7d32dc428089b89ef106618706 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting consensus metadata
I20260926 19:27:49.627694 9299 tablet_replica.cc:330] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: stopping tablet replica
I20260926 19:27:49.628096 9299 raft_consensus.cc:2229] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:49.628537 9299 raft_consensus.cc:2258] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:49.629640 8749 catalog_manager.cc:4841] TS 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651): tablet 24b23c7d32dc428089b89ef106618706 (table test-table [id=abe9f92ce26f4c05bea17c547a190c16]) successfully deleted
I20260926 19:27:49.630183 9180 tablet_service.cc:1467] Processing CreateTablet for tablet 887185d319d341e28ebdb7145bd6faf8 (DEFAULT_TABLE table=test-table [id=077b43bcdb8d422b915948351db9e919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:49.630559 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 887185d319d341e28ebdb7145bd6faf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.634130 9276 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Bootstrap starting.
I20260926 19:27:49.635831 9276 tablet_bootstrap.cc:654] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.636538 9275 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Bootstrap starting.
I20260926 19:27:49.638003 9275 tablet_bootstrap.cc:654] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.638082 9276 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: No bootstrap required, opened a new log
I20260926 19:27:49.638224 9276 ts_tablet_manager.cc:1397] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 19:27:49.638873 9299 ts_tablet_manager.cc:1905] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:49.639011 9276 raft_consensus.cc:348] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.639219 9276 raft_consensus.cc:374] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.639317 9276 raft_consensus.cc:729] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.639597 9276 consensus_queue.cc:260] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.642426 9276 ts_tablet_manager.cc:1428] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:27:49.645560 9299 ts_tablet_manager.cc:1918] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:49.645713 9299 log.cc:1196] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/24b23c7d32dc428089b89ef106618706
I20260926 19:27:49.646263 9274 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Bootstrap starting.
I20260926 19:27:49.646317 9275 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: No bootstrap required, opened a new log
I20260926 19:27:49.646422 9299 ts_tablet_manager.cc:1939] T 24b23c7d32dc428089b89ef106618706 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting consensus metadata
I20260926 19:27:49.646441 9275 ts_tablet_manager.cc:1397] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 19:27:49.647336 9275 raft_consensus.cc:348] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.647557 9275 raft_consensus.cc:374] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.647643 9275 raft_consensus.cc:729] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.648006 9274 tablet_bootstrap.cc:654] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.647976 9275 consensus_queue.cc:260] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.648172 9301 ts_tablet_manager.cc:1905] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:49.648173 8751 catalog_manager.cc:4841] TS 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919): tablet 24b23c7d32dc428089b89ef106618706 (table test-table [id=abe9f92ce26f4c05bea17c547a190c16]) successfully deleted
I20260926 19:27:49.648590 9275 ts_tablet_manager.cc:1428] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 19:27:49.653455 9274 tablet_bootstrap.cc:492] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: No bootstrap required, opened a new log
I20260926 19:27:49.653656 9274 ts_tablet_manager.cc:1397] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260926 19:27:49.654466 9274 raft_consensus.cc:348] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.654749 9274 raft_consensus.cc:374] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.654896 9274 raft_consensus.cc:729] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc227cad664fc5a6ba653eebfa1ce0, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.655234 9301 ts_tablet_manager.cc:1918] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:49.655388 9301 log.cc:1196] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/24b23c7d32dc428089b89ef106618706
I20260926 19:27:49.655238 9274 consensus_queue.cc:260] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.655928 9301 ts_tablet_manager.cc:1939] T 24b23c7d32dc428089b89ef106618706 P 944eb67c51a042a08dfe1d89c64ba304: Deleting consensus metadata
I20260926 19:27:49.657424 8751 catalog_manager.cc:4841] TS 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969): tablet 24b23c7d32dc428089b89ef106618706 (table test-table [id=abe9f92ce26f4c05bea17c547a190c16]) successfully deleted
I20260926 19:27:49.657824 9274 ts_tablet_manager.cc:1428] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 19:27:49.680660 9297 raft_consensus.cc:482] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:49.680909 9297 raft_consensus.cc:504] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.681896 9297 leader_election.cc:288] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:49.682435 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887185d319d341e28ebdb7145bd6faf8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" is_pre_election: true
I20260926 19:27:49.682519 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887185d319d341e28ebdb7145bd6faf8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502" is_pre_election: true
I20260926 19:27:49.682710 8927 raft_consensus.cc:2454] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:49.682783 9205 raft_consensus.cc:2454] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:49.683166 8994 leader_election.cc:302] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.683472 9297 raft_consensus.cc:2781] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:49.683612 9297 raft_consensus.cc:482] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:49.683743 9297 raft_consensus.cc:3037] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.685988 9297 raft_consensus.cc:504] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.686559 9297 leader_election.cc:288] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:49.687016 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887185d319d341e28ebdb7145bd6faf8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
I20260926 19:27:49.687078 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "887185d319d341e28ebdb7145bd6faf8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502"
I20260926 19:27:49.687203 8927 raft_consensus.cc:3037] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.687274 9205 raft_consensus.cc:3037] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.689119 8927 raft_consensus.cc:2454] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:49.689515 8994 leader_election.cc:302] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.689777 9297 raft_consensus.cc:2781] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:49.689950 9297 raft_consensus.cc:686] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Becoming Leader. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Running, Role: LEADER
I20260926 19:27:49.690249 9205 raft_consensus.cc:2454] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:49.690222 9297 consensus_queue.cc:237] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.692840 8765 catalog_manager.cc:5495] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2). New cstate: current_term: 1 leader_uuid: "944eb67c51a042a08dfe1d89c64ba304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } health_report { overall_health: UNKNOWN } } }
Sep 26 19:27:49 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450867, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.62@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:49.740157 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.723743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.1.39.62:37115)
0926 19:27:49.724010 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.724020 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.724512 (+ 492us) server_negotiation.cc:407] Connection header received
0926 19:27:49.724693 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.724698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.724774 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.724857 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:49.726370 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.727127 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.729022 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.729580 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.729702 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.733928 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:49.733934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:49.733938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:49.733971 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:49.736806 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.738007 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.738012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.738014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.738132 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:49.738553 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:49.738556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:49.738558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:49.738811 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:49.738916 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.739456 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.739762 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":114}
I20260926 19:27:49.740206 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.723734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37115 (local address 127.0.0.1:56046)
0926 19:27:49.724382 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.724403 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.724558 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.724913 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.724918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.725875 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:49.726228 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.726240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.727264 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.727271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.728853 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.728868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.729767 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.729774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.729983 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.730752 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:49.730776 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:49.733678 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:49.736980 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.736994 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.737003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.737813 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.738246 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:49.738251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:49.738255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:49.738427 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:49.738935 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:49.738943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:49.739160 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.739334 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.739648 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
I20260926 19:27:49.745826 8765 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:56046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 19:27:49.747076 8765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:49.754211 8901 tablet_service.cc:1467] Processing CreateTablet for tablet d2539302e3e3499c9acbd148daa729c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe7579259e84c018828150702cc0c19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:49.754781 8901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2539302e3e3499c9acbd148daa729c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.754699 9041 tablet_service.cc:1467] Processing CreateTablet for tablet d2539302e3e3499c9acbd148daa729c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe7579259e84c018828150702cc0c19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:49.755236 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2539302e3e3499c9acbd148daa729c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.757864 9180 tablet_service.cc:1467] Processing CreateTablet for tablet d2539302e3e3499c9acbd148daa729c2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1fe7579259e84c018828150702cc0c19]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 19:27:49.758524 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2539302e3e3499c9acbd148daa729c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:49.762239 9274 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: Bootstrap starting.
I20260926 19:27:49.763859 9274 tablet_bootstrap.cc:654] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.764871 9276 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: Bootstrap starting.
I20260926 19:27:49.766268 9276 tablet_bootstrap.cc:654] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.767903 9275 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: Bootstrap starting.
I20260926 19:27:49.768447 9274 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: No bootstrap required, opened a new log
I20260926 19:27:49.768603 9274 ts_tablet_manager.cc:1397] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 19:27:49.769217 9274 raft_consensus.cc:348] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.769481 9274 raft_consensus.cc:374] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.769562 9274 raft_consensus.cc:729] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc227cad664fc5a6ba653eebfa1ce0, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.769881 9275 tablet_bootstrap.cc:654] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:49.769785 9274 consensus_queue.cc:260] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.770382 9274 ts_tablet_manager.cc:1428] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:27:49.771068 9282 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.771499 9282 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.772332 9276 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: No bootstrap required, opened a new log
I20260926 19:27:49.772518 9276 ts_tablet_manager.cc:1397] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 19:27:49.773221 9276 raft_consensus.cc:348] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.773496 9276 raft_consensus.cc:374] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.773587 9276 raft_consensus.cc:729] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.773882 9276 consensus_queue.cc:260] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.774493 9276 ts_tablet_manager.cc:1428] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:27:49.774371 9297 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.774639 9297 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.774813 9275 tablet_bootstrap.cc:492] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: No bootstrap required, opened a new log
I20260926 19:27:49.774981 9275 ts_tablet_manager.cc:1397] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260926 19:27:49.775794 9275 raft_consensus.cc:348] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.776458 9275 raft_consensus.cc:374] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:49.776551 9275 raft_consensus.cc:729] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Initialized, Role: FOLLOWER
I20260926 19:27:49.777022 9275 consensus_queue.cc:260] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.777813 9275 ts_tablet_manager.cc:1428] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:49.777635 9281 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.778055 9281 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.842396 9281 raft_consensus.cc:482] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:49.842658 9281 raft_consensus.cc:504] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.843951 9281 leader_election.cc:288] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969)
W20260926 19:27:49.849984 9251 tablet.cc:2369] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:27:49.852026 9269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.844344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:51511 (local address 127.1.39.1:33919)
0926 19:27:49.844546 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.844554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.844808 (+ 254us) server_negotiation.cc:407] Connection header received
0926 19:27:49.845116 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.845125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.845246 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.845520 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.846162 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.847315 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.849725 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.850970 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.851107 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.851208 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.851593 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.851735 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:27:49.852154 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.845061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:40297 (local address 127.1.39.2:37969)
0926 19:27:49.845445 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.845451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.845589 (+ 138us) server_negotiation.cc:407] Connection header received
0926 19:27:49.845723 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.845728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.845797 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.845916 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:27:49.846424 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.847583 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.849633 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.850969 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.851108 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.851205 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.851718 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.851843 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260926 19:27:49.852079 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.844261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33919 (local address 127.1.39.3:51511)
0926 19:27:49.844706 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.844721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.844917 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.845730 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.845735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.845765 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.845948 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.845957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.847516 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.847521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.849592 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.849607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.851146 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.851153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.851411 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.851446 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.851551 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.851701 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":33,"thread_start_us":244,"threads_started":1}
I20260926 19:27:49.852886 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.844907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:37969 (local address 127.1.39.3:40297)
0926 19:27:49.845477 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.845504 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.845610 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.845917 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.845922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.845944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:27:49.846250 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.846263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.847702 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.847710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.849489 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.849502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.851374 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.851378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.851499 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.851556 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.852383 (+ 827us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.852532 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":25,"thread_start_us":294,"threads_started":1}
I20260926 19:27:49.853061 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2539302e3e3499c9acbd148daa729c2" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" is_pre_election: true
I20260926 19:27:49.853349 8927 raft_consensus.cc:2454] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 0.
I20260926 19:27:49.853860 9133 leader_election.cc:302] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.854275 9281 raft_consensus.cc:2781] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:49.854403 9281 raft_consensus.cc:482] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:49.854511 9281 raft_consensus.cc:3037] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.854280 9061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2539302e3e3499c9acbd148daa729c2" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944eb67c51a042a08dfe1d89c64ba304" is_pre_election: true
I20260926 19:27:49.854746 9061 raft_consensus.cc:2454] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 0.
I20260926 19:27:49.856896 9281 raft_consensus.cc:504] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.857584 9281 leader_election.cc:288] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969)
I20260926 19:27:49.857944 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2539302e3e3499c9acbd148daa729c2" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
I20260926 19:27:49.858093 9061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2539302e3e3499c9acbd148daa729c2" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944eb67c51a042a08dfe1d89c64ba304"
I20260926 19:27:49.858178 8927 raft_consensus.cc:3037] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.858306 9061 raft_consensus.cc:3037] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:49.860257 8927 raft_consensus.cc:2454] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 1.
I20260926 19:27:49.860612 9133 leader_election.cc:302] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:49.860890 9281 raft_consensus.cc:2781] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:49.861438 9281 raft_consensus.cc:686] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 LEADER]: Becoming Leader. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Running, Role: LEADER
I20260926 19:27:49.861696 9061 raft_consensus.cc:2454] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 1.
I20260926 19:27:49.861851 9281 consensus_queue.cc:237] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:49.864712 9317 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: TxnStatusTablet state changed. Reason: New leader 14c4c76aaf3a41bcaef3e0700e451502. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.865015 9317 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:49.865649 9319 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 19:27:49.865911 9319 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 19:27:49.865481 8763 catalog_manager.cc:5495] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:49.907119 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.899304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35651 (local address 127.0.0.1:58762)
0926 19:27:49.899566 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.899587 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.899723 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.900335 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.900340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.900353 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:49.900561 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.900572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.902206 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.902212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.903893 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.903907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.904882 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.904890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.905147 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.905258 (+ 111us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:49.906521 (+ 1263us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:49.906543 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.906631 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.906775 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 19:27:49.907143 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.899673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.1.39.3:35651)
0926 19:27:49.899860 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.899866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.899887 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:49.900001 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.900007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.900132 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.900296 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:49.900721 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.902070 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.904056 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.904709 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.904845 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.905525 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:49.906256 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:49.906465 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.906731 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.906816 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:27:49.912055 8927 raft_consensus.cc:1261] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Refusing update from remote peer 14c4c76aaf3a41bcaef3e0700e451502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.912817 9317 consensus_queue.cc:1035] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:49.916106 9282 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: TxnStatusTablet state changed. Reason: New leader 14c4c76aaf3a41bcaef3e0700e451502. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.916393 9282 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.918622 9317 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.918865 9317 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:49.921049 9282 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.921271 9282 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 9dcc227cad664fc5a6ba653eebfa1ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.926012 9061 raft_consensus.cc:1261] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Refusing update from remote peer 14c4c76aaf3a41bcaef3e0700e451502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.926766 9281 consensus_queue.cc:1035] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:49.931672 9285 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.932044 9285 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.932054 9297 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: TxnStatusTablet state changed. Reason: New leader 14c4c76aaf3a41bcaef3e0700e451502. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.934207 9297 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 944eb67c51a042a08dfe1d89c64ba304: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 19:27:49.950944 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.943176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:57555 (local address 127.1.39.3:35651)
0926 19:27:49.943370 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:49.943376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:49.944131 (+ 755us) server_negotiation.cc:407] Connection header received
0926 19:27:49.944353 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:49.944358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:49.944441 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:49.944538 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:49.945215 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.946433 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.948650 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.949441 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.949576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.950141 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:49.950284 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:49.950381 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:49.950556 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 19:27:49.950642 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:27:49.951311 9326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:49.943216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35651 (local address 127.1.39.2:57555)
0926 19:27:49.944009 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:49.944029 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:49.944191 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:49.944574 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:49.944578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:49.944591 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:49.945020 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.945033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.946556 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.946562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.948493 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:49.948508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:49.949651 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:49.949658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:49.949854 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:49.949938 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:49.950406 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:49.950417 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:27:49.950522 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 19:27:49.950664 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":427,"threads_started":1}
I20260926 19:27:49.955256 9281 tablet_replica.cc:439] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } } }
I20260926 19:27:49.955586 9281 tablet_replica.cc:442] T d2539302e3e3499c9acbd148daa729c2 P 14c4c76aaf3a41bcaef3e0700e451502: This TxnStatusTablet replica's current role is: LEADER
I20260926 19:27:49.965458 8926 raft_consensus.cc:1261] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.965525 9205 raft_consensus.cc:1261] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:49.966089 9297 consensus_queue.cc:1035] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:49.966341 9285 consensus_queue.cc:1035] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:50.025053 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.012807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:39487 (local address 127.1.39.2:37969)
0926 19:27:50.013000 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.013007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.017906 (+ 4899us) server_negotiation.cc:407] Connection header received
0926 19:27:50.018226 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.018232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.018324 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.018449 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:50.019328 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.020483 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.022753 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.023448 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.023575 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.024237 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:50.024374 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:50.024433 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.024656 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.024735 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:27:50.025857 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.013024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:37969 (local address 127.1.39.3:39487)
0926 19:27:50.017802 (+ 4778us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:50.017827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:50.018032 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:50.018627 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:50.018634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:50.018646 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 19:27:50.019140 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:50.019153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.020682 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.020687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:50.022595 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:50.022610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.023725 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.023732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:50.023930 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.024012 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 19:27:50.024518 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 19:27:50.024534 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 19:27:50.025254 (+ 720us) client_negotiation.cc:240] Negotiation successful
0926 19:27:50.025414 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4556,"thread_start_us":4436,"threads_started":1}
I20260926 19:27:50.069074 8764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.069404 8764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.072719 8764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Sending DeleteTablet for 3 replicas of tablet 887185d319d341e28ebdb7145bd6faf8
I20260926 19:27:50.073553 8901 tablet_service.cc:1514] Processing DeleteTablet for tablet 887185d319d341e28ebdb7145bd6faf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48696
I20260926 19:27:50.073833 9299 tablet_replica.cc:330] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: stopping tablet replica
I20260926 19:27:50.074033 9299 raft_consensus.cc:2229] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.074225 9299 raft_consensus.cc:2258] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.074692 9041 tablet_service.cc:1514] Processing DeleteTablet for tablet 887185d319d341e28ebdb7145bd6faf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:41396
I20260926 19:27:50.074950 9301 tablet_replica.cc:330] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: stopping tablet replica
I20260926 19:27:50.075117 9301 raft_consensus.cc:2229] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:50.075424 9301 raft_consensus.cc:2258] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.075555 9299 ts_tablet_manager.cc:1905] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.080298 9180 tablet_service.cc:1514] Processing DeleteTablet for tablet 887185d319d341e28ebdb7145bd6faf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260926 19:27:50.080595 9300 tablet_replica.cc:330] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: stopping tablet replica
I20260926 19:27:50.080801 9300 raft_consensus.cc:2229] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.080988 9300 raft_consensus.cc:2258] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.081246 9301 ts_tablet_manager.cc:1905] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.082182 9300 ts_tablet_manager.cc:1905] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.088552 9301 ts_tablet_manager.cc:1918] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.088735 9301 log.cc:1196] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/887185d319d341e28ebdb7145bd6faf8
I20260926 19:27:50.089180 9301 ts_tablet_manager.cc:1939] T 887185d319d341e28ebdb7145bd6faf8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting consensus metadata
I20260926 19:27:50.089195 9300 ts_tablet_manager.cc:1918] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.090646 9299 ts_tablet_manager.cc:1918] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.090787 9299 log.cc:1196] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/887185d319d341e28ebdb7145bd6faf8
I20260926 19:27:50.091163 9299 ts_tablet_manager.cc:1939] T 887185d319d341e28ebdb7145bd6faf8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting consensus metadata
I20260926 19:27:50.092227 8751 catalog_manager.cc:4841] TS 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919): tablet 887185d319d341e28ebdb7145bd6faf8 (table test-table [id=077b43bcdb8d422b915948351db9e919]) successfully deleted
I20260926 19:27:50.092520 9300 log.cc:1196] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/887185d319d341e28ebdb7145bd6faf8
I20260926 19:27:50.092958 9300 ts_tablet_manager.cc:1939] T 887185d319d341e28ebdb7145bd6faf8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting consensus metadata
I20260926 19:27:50.094010 8749 catalog_manager.cc:4841] TS 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651): tablet 887185d319d341e28ebdb7145bd6faf8 (table test-table [id=077b43bcdb8d422b915948351db9e919]) successfully deleted
I20260926 19:27:50.098431 8751 catalog_manager.cc:4841] TS 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969): tablet 887185d319d341e28ebdb7145bd6faf8 (table test-table [id=077b43bcdb8d422b915948351db9e919]) successfully deleted
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:27:50.160434 8860 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:50.148661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.1.39.62:37115)
0926 19:27:50.148930 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.148937 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.149356 (+ 419us) server_negotiation.cc:407] Connection header received
0926 19:27:50.149554 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.149558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.149623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.149710 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:50.151465 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.152229 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.154042 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.154605 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.154717 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.160147 (+ 5430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56058: BlockingRecv error: recv got EOF from 127.0.0.1:56058 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM
I20260926 19:27:50.182175 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.166411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.1.39.62:37115)
0926 19:27:50.166570 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.166574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.167064 (+ 490us) server_negotiation.cc:407] Connection header received
0926 19:27:50.167248 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.167252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.167336 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.167434 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:50.168983 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.169802 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.171660 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.172076 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.172173 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.176768 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:50.176773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:50.176777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:50.176812 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:50.179477 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.180166 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.180176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.180178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.180307 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.180705 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.180715 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.180717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.180985 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:50.181243 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.181599 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.181821 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 19:27:50.185837 8764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:50.186466 8764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:50.193414 8901 tablet_service.cc:1467] Processing CreateTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 (DEFAULT_TABLE table=test-table [id=936d33f4192b41a0a84ad7534c7d0534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.193905 8901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e33dbcc96b48889af0c46cf58fbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.193821 9180 tablet_service.cc:1467] Processing CreateTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 (DEFAULT_TABLE table=test-table [id=936d33f4192b41a0a84ad7534c7d0534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.194262 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e33dbcc96b48889af0c46cf58fbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.197733 9041 tablet_service.cc:1467] Processing CreateTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 (DEFAULT_TABLE table=test-table [id=936d33f4192b41a0a84ad7534c7d0534]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.198136 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e33dbcc96b48889af0c46cf58fbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.199267 9274 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Bootstrap starting.
I20260926 19:27:50.200876 9274 tablet_bootstrap.cc:654] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.203181 9276 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Bootstrap starting.
I20260926 19:27:50.204356 9275 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Bootstrap starting.
I20260926 19:27:50.204736 9276 tablet_bootstrap.cc:654] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.206266 9275 tablet_bootstrap.cc:654] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.215925 9276 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: No bootstrap required, opened a new log
I20260926 19:27:50.215981 9275 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: No bootstrap required, opened a new log
I20260926 19:27:50.216113 9276 ts_tablet_manager.cc:1397] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260926 19:27:50.216149 9275 ts_tablet_manager.cc:1397] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260926 19:27:50.216730 9276 raft_consensus.cc:348] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.216948 9276 raft_consensus.cc:374] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.217043 9276 raft_consensus.cc:729] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.216982 9275 raft_consensus.cc:348] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.217283 9275 raft_consensus.cc:374] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.217424 9275 raft_consensus.cc:729] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.217317 9276 consensus_queue.cc:260] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.217706 9275 consensus_queue.cc:260] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.219317 9275 ts_tablet_manager.cc:1428] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:50.220355 9276 ts_tablet_manager.cc:1428] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 19:27:50.220775 9274 tablet_bootstrap.cc:492] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: No bootstrap required, opened a new log
I20260926 19:27:50.220930 9274 ts_tablet_manager.cc:1397] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.007s
I20260926 19:27:50.221714 9274 raft_consensus.cc:348] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.221959 9274 raft_consensus.cc:374] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.222045 9274 raft_consensus.cc:729] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc227cad664fc5a6ba653eebfa1ce0, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.222270 9274 consensus_queue.cc:260] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.222788 9274 ts_tablet_manager.cc:1428] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 19:27:50.276638 9327 raft_consensus.cc:482] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:50.276894 9327 raft_consensus.cc:504] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.277849 9327 leader_election.cc:288] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969)
I20260926 19:27:50.278417 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e33dbcc96b48889af0c46cf58fbd96" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" is_pre_election: true
I20260926 19:27:50.278508 9060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e33dbcc96b48889af0c46cf58fbd96" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944eb67c51a042a08dfe1d89c64ba304" is_pre_election: true
I20260926 19:27:50.278707 8927 raft_consensus.cc:2454] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 0.
I20260926 19:27:50.278775 9060 raft_consensus.cc:2454] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 0.
I20260926 19:27:50.279203 9133 leader_election.cc:302] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:50.279470 9327 raft_consensus.cc:2781] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:50.279603 9327 raft_consensus.cc:482] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:50.279731 9327 raft_consensus.cc:3037] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.282047 9327 raft_consensus.cc:504] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.282631 9327 leader_election.cc:288] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969)
I20260926 19:27:50.283070 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e33dbcc96b48889af0c46cf58fbd96" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
I20260926 19:27:50.283116 9060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e33dbcc96b48889af0c46cf58fbd96" candidate_uuid: "14c4c76aaf3a41bcaef3e0700e451502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944eb67c51a042a08dfe1d89c64ba304"
I20260926 19:27:50.283291 9060 raft_consensus.cc:3037] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.283300 8927 raft_consensus.cc:3037] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.286128 8927 raft_consensus.cc:2454] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 1.
I20260926 19:27:50.286144 9060 raft_consensus.cc:2454] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c4c76aaf3a41bcaef3e0700e451502 in term 1.
I20260926 19:27:50.286505 9133 leader_election.cc:302] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c4c76aaf3a41bcaef3e0700e451502, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:50.286857 9327 raft_consensus.cc:2781] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:50.287040 9327 raft_consensus.cc:686] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 LEADER]: Becoming Leader. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Running, Role: LEADER
I20260926 19:27:50.287300 9327 consensus_queue.cc:237] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.289913 8764 catalog_manager.cc:5495] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "14c4c76aaf3a41bcaef3e0700e451502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } health_report { overall_health: HEALTHY } } }
I20260926 19:27:50.321952 9271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.314632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.1.39.3:35651)
0926 19:27:50.314830 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.314838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.314859 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:50.314955 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.314962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.315047 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.315168 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:50.315533 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.317134 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.319893 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.320589 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.320705 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.321174 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:50.321337 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:50.321420 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.321573 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.321652 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:27:50.325340 9060 raft_consensus.cc:1261] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Refusing update from remote peer 14c4c76aaf3a41bcaef3e0700e451502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:50.325428 8927 raft_consensus.cc:1261] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Refusing update from remote peer 14c4c76aaf3a41bcaef3e0700e451502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:50.326061 9281 consensus_queue.cc:1035] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:50.326509 9327 consensus_queue.cc:1035] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:50.348134 9290 mvcc.cc:205] Tried to move back new op lower bound from 7333686764846702592 to 7333686764698894336. Current Snapshot: MvccSnapshot[applied={T|T < 7333686764846702592}]
I20260926 19:27:50.353585 8765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.353813 8765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.357493 8765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Sending DeleteTablet for 3 replicas of tablet 53e33dbcc96b48889af0c46cf58fbd96
I20260926 19:27:50.358819 9041 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:41396
I20260926 19:27:50.359082 9301 tablet_replica.cc:330] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: stopping tablet replica
I20260926 19:27:50.359247 9301 raft_consensus.cc:2229] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.359316 8901 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48696
I20260926 19:27:50.359462 9301 raft_consensus.cc:2258] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.359782 9299 tablet_replica.cc:330] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: stopping tablet replica
I20260926 19:27:50.359954 9299 raft_consensus.cc:2229] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.360154 9299 raft_consensus.cc:2258] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.361388 9299 ts_tablet_manager.cc:1905] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.361542 9180 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e33dbcc96b48889af0c46cf58fbd96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260926 19:27:50.361825 9300 tablet_replica.cc:330] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: stopping tablet replica
I20260926 19:27:50.362015 9300 raft_consensus.cc:2229] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:50.362499 9300 raft_consensus.cc:2258] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.363514 8764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:27:50.364140 8764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:27:50.366108 9301 ts_tablet_manager.cc:1905] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.366132 9300 ts_tablet_manager.cc:1905] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.368078 9299 ts_tablet_manager.cc:1918] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:50.368208 9299 log.cc:1196] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/53e33dbcc96b48889af0c46cf58fbd96
I20260926 19:27:50.368655 9299 ts_tablet_manager.cc:1939] T 53e33dbcc96b48889af0c46cf58fbd96 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting consensus metadata
I20260926 19:27:50.369640 8751 catalog_manager.cc:4841] TS 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919): tablet 53e33dbcc96b48889af0c46cf58fbd96 (table test-table [id=936d33f4192b41a0a84ad7534c7d0534]) successfully deleted
I20260926 19:27:50.371609 9300 ts_tablet_manager.cc:1918] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:50.371762 9300 log.cc:1196] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/53e33dbcc96b48889af0c46cf58fbd96
I20260926 19:27:50.371994 9301 ts_tablet_manager.cc:1918] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:27:50.372117 9301 log.cc:1196] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/53e33dbcc96b48889af0c46cf58fbd96
I20260926 19:27:50.372213 9300 ts_tablet_manager.cc:1939] T 53e33dbcc96b48889af0c46cf58fbd96 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting consensus metadata
I20260926 19:27:50.372514 9301 ts_tablet_manager.cc:1939] T 53e33dbcc96b48889af0c46cf58fbd96 P 944eb67c51a042a08dfe1d89c64ba304: Deleting consensus metadata
I20260926 19:27:50.373167 8749 catalog_manager.cc:4841] TS 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651): tablet 53e33dbcc96b48889af0c46cf58fbd96 (table test-table [id=936d33f4192b41a0a84ad7534c7d0534]) successfully deleted
I20260926 19:27:50.373431 8751 catalog_manager.cc:4841] TS 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969): tablet 53e33dbcc96b48889af0c46cf58fbd96 (table test-table [id=936d33f4192b41a0a84ad7534c7d0534]) successfully deleted
I20260926 19:27:50.377483 8901 tablet_service.cc:1467] Processing CreateTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (DEFAULT_TABLE table=test-table [id=d6207a5deeca459a8170b906cfab578a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.377867 8901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e48fc923d54d6cbf8b9147d1fab8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.377929 9180 tablet_service.cc:1467] Processing CreateTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (DEFAULT_TABLE table=test-table [id=d6207a5deeca459a8170b906cfab578a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.378326 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e48fc923d54d6cbf8b9147d1fab8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.384303 9041 tablet_service.cc:1467] Processing CreateTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (DEFAULT_TABLE table=test-table [id=d6207a5deeca459a8170b906cfab578a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:27:50.384445 9275 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Bootstrap starting.
I20260926 19:27:50.384734 9041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e48fc923d54d6cbf8b9147d1fab8a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:50.386121 9275 tablet_bootstrap.cc:654] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.392813 9274 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Bootstrap starting.
I20260926 19:27:50.395059 9274 tablet_bootstrap.cc:654] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.397084 9276 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Bootstrap starting.
I20260926 19:27:50.398636 9274 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: No bootstrap required, opened a new log
I20260926 19:27:50.398968 9276 tablet_bootstrap.cc:654] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:50.398805 9274 ts_tablet_manager.cc:1397] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 19:27:50.400434 9274 raft_consensus.cc:348] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.400766 9274 raft_consensus.cc:374] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.400868 9274 raft_consensus.cc:729] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dcc227cad664fc5a6ba653eebfa1ce0, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.401209 9274 consensus_queue.cc:260] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.401973 9274 ts_tablet_manager.cc:1428] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 19:27:50.407593 9276 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: No bootstrap required, opened a new log
I20260926 19:27:50.407768 9276 ts_tablet_manager.cc:1397] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260926 19:27:50.408368 9276 raft_consensus.cc:348] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.408608 9276 raft_consensus.cc:374] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.408686 9276 raft_consensus.cc:729] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.408915 9276 consensus_queue.cc:260] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.410045 9276 ts_tablet_manager.cc:1428] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260926 19:27:50.411425 9275 tablet_bootstrap.cc:492] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: No bootstrap required, opened a new log
I20260926 19:27:50.411594 9275 ts_tablet_manager.cc:1397] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.012s
I20260926 19:27:50.412424 9275 raft_consensus.cc:348] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.412705 9275 raft_consensus.cc:374] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:50.412806 9275 raft_consensus.cc:729] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c4c76aaf3a41bcaef3e0700e451502, State: Initialized, Role: FOLLOWER
I20260926 19:27:50.413094 9275 consensus_queue.cc:260] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.413694 9275 ts_tablet_manager.cc:1428] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 19:27:50.416445 9280 raft_consensus.cc:482] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:27:50.416592 9280 raft_consensus.cc:504] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.417248 9280 leader_election.cc:288] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:50.417672 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e48fc923d54d6cbf8b9147d1fab8a8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" is_pre_election: true
I20260926 19:27:50.417933 8927 raft_consensus.cc:2454] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:50.417848 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e48fc923d54d6cbf8b9147d1fab8a8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502" is_pre_election: true
I20260926 19:27:50.418092 9205 raft_consensus.cc:2454] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 0.
I20260926 19:27:50.418316 8994 leader_election.cc:302] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:50.418716 9280 raft_consensus.cc:2781] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:27:50.418808 9280 raft_consensus.cc:482] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:27:50.418921 9280 raft_consensus.cc:3037] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.420902 9280 raft_consensus.cc:504] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.421466 9280 leader_election.cc:288] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Requested vote from peers 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919), 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651)
I20260926 19:27:50.421806 8927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e48fc923d54d6cbf8b9147d1fab8a8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0"
I20260926 19:27:50.422013 8927 raft_consensus.cc:3037] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.422044 9205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e48fc923d54d6cbf8b9147d1fab8a8" candidate_uuid: "944eb67c51a042a08dfe1d89c64ba304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14c4c76aaf3a41bcaef3e0700e451502"
I20260926 19:27:50.422248 9205 raft_consensus.cc:3037] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:50.423938 8927 raft_consensus.cc:2454] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:50.424279 8994 leader_election.cc:302] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944eb67c51a042a08dfe1d89c64ba304, 9dcc227cad664fc5a6ba653eebfa1ce0; no voters:
I20260926 19:27:50.424551 9280 raft_consensus.cc:2781] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:50.424711 9280 raft_consensus.cc:686] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Becoming Leader. State: Replica: 944eb67c51a042a08dfe1d89c64ba304, State: Running, Role: LEADER
I20260926 19:27:50.424952 9280 consensus_queue.cc:237] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } }
I20260926 19:27:50.425242 9205 raft_consensus.cc:2454] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944eb67c51a042a08dfe1d89c64ba304 in term 1.
I20260926 19:27:50.427444 8764 catalog_manager.cc:5495] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2). New cstate: current_term: 1 leader_uuid: "944eb67c51a042a08dfe1d89c64ba304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944eb67c51a042a08dfe1d89c64ba304" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 37969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 } health_report { overall_health: UNKNOWN } } }
I20260926 19:27:50.454228 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.449508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.1.39.2:37969)
0926 19:27:50.449652 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.449660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.449742 (+ 82us) server_negotiation.cc:407] Connection header received
0926 19:27:50.449951 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.449955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.450029 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.450133 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:27:50.450516 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.451227 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.452534 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.452952 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.453083 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.453575 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:27:50.453673 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:27:50.453769 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.453881 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.453956 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:27:50.462107 9205 raft_consensus.cc:1261] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:50.462617 8927 raft_consensus.cc:1261] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Refusing update from remote peer 944eb67c51a042a08dfe1d89c64ba304: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:27:50.462905 9285 consensus_queue.cc:1035] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14c4c76aaf3a41bcaef3e0700e451502" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:50.463629 9285 consensus_queue.cc:1035] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dcc227cad664fc5a6ba653eebfa1ce0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:27:50.516080 8762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.516311 8762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:27:50.519101 8762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 214cf3628c604becb061f1e7c65395b3: Sending DeleteTablet for 3 replicas of tablet d1e48fc923d54d6cbf8b9147d1fab8a8
I20260926 19:27:50.519912 8901 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48696
I20260926 19:27:50.519951 9180 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58748
I20260926 19:27:50.520088 9041 tablet_service.cc:1514] Processing DeleteTablet for tablet d1e48fc923d54d6cbf8b9147d1fab8a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:27:50 UTC) from {username='oryx', principal='oryx/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:41396
I20260926 19:27:50.520210 9299 tablet_replica.cc:330] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: stopping tablet replica
I20260926 19:27:50.520226 9300 tablet_replica.cc:330] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: stopping tablet replica
I20260926 19:27:50.520397 9299 raft_consensus.cc:2229] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.520406 9300 raft_consensus.cc:2229] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:27:50.520581 9299 raft_consensus.cc:2258] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.520601 9300 raft_consensus.cc:2258] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.520854 9301 tablet_replica.cc:330] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: stopping tablet replica
I20260926 19:27:50.521080 9301 raft_consensus.cc:2229] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:27:50.521384 9301 raft_consensus.cc:2258] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:27:50.521718 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8836
I20260926 19:27:50.522658 9300 ts_tablet_manager.cc:1905] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.522984 9299 ts_tablet_manager.cc:1905] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.529595 9300 ts_tablet_manager.cc:1918] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.529763 9300 log.cc:1196] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/d1e48fc923d54d6cbf8b9147d1fab8a8
I20260926 19:27:50.530224 9300 ts_tablet_manager.cc:1939] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 14c4c76aaf3a41bcaef3e0700e451502: Deleting consensus metadata
I20260926 19:27:50.531237 9299 ts_tablet_manager.cc:1918] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.531289 8749 catalog_manager.cc:4841] TS 14c4c76aaf3a41bcaef3e0700e451502 (127.1.39.3:35651): tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (table test-table [id=d6207a5deeca459a8170b906cfab578a]) successfully deleted
I20260926 19:27:50.531737 9299 log.cc:1196] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/d1e48fc923d54d6cbf8b9147d1fab8a8
I20260926 19:27:50.532310 9299 ts_tablet_manager.cc:1939] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 9dcc227cad664fc5a6ba653eebfa1ce0: Deleting consensus metadata
I20260926 19:27:50.532915 9301 ts_tablet_manager.cc:1905] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:27:50.533666 8751 catalog_manager.cc:4841] TS 9dcc227cad664fc5a6ba653eebfa1ce0 (127.1.39.1:33919): tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (table test-table [id=d6207a5deeca459a8170b906cfab578a]) successfully deleted
I20260926 19:27:50.539332 9301 ts_tablet_manager.cc:1918] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 19:27:50.539482 9301 log.cc:1196] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/d1e48fc923d54d6cbf8b9147d1fab8a8
I20260926 19:27:50.539886 9301 ts_tablet_manager.cc:1939] T d1e48fc923d54d6cbf8b9147d1fab8a8 P 944eb67c51a042a08dfe1d89c64ba304: Deleting consensus metadata
I20260926 19:27:50.540932 8751 catalog_manager.cc:4841] TS 944eb67c51a042a08dfe1d89c64ba304 (127.1.39.2:37969): tablet d1e48fc923d54d6cbf8b9147d1fab8a8 (table test-table [id=d6207a5deeca459a8170b906cfab578a]) successfully deleted
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.1@KRBTEST.COM
I20260926 19:27:50.552309 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.523689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.1.39.1:33919)
0926 19:27:50.531881 (+ 8192us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.531887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.537118 (+ 5231us) server_negotiation.cc:407] Connection header received
0926 19:27:50.537390 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.537395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.537492 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.537559 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:50.538888 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.539774 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.541135 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.541667 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.541796 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.546171 (+ 4375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:50.546185 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:50.546203 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:50.546239 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:50.549583 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.550375 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.550382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.550384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.550525 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.550939 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.550945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.550947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.551276 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:50.551558 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.551830 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.551993 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8074,"thread_start_us":7980,"threads_started":1}
I20260926 19:27:50.553005 8967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:50.661798 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8836
I20260926 19:27:50.680917 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8976
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.2@KRBTEST.COM
I20260926 19:27:50.697495 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.681742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.1.39.2:37969)
0926 19:27:50.681977 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.681982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.682006 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:27:50.682133 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.682139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.682226 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.682346 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:50.683581 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.684647 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.686012 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.686632 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.686781 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.690846 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:50.690864 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:50.690886 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:50.690932 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:50.694727 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.695407 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.695417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.695421 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.695580 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.695911 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.695916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.695920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.696388 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:50.696702 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.696877 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.697119 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 19:27:50.698338 9106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:50.823277 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8976
I20260926 19:27:50.846148 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9115
Sep 26 19:27:50 dist-test-slave-9wb0 krb5kdc[8720](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.39.3@KRBTEST.COM
I20260926 19:27:50.861344 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:50.846901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.1.39.3:35651)
0926 19:27:50.847413 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:50.847417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:50.847433 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:27:50.847487 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:50.847491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:50.847550 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:50.847641 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:50.848854 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.849812 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.851045 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:50.851437 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:50.851530 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:50.855910 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:50.855926 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:50.855943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:50.855980 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:50.858794 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.859392 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.859399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.859401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.859530 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:50.859860 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:50.859863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:50.859865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:50.860212 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:50.860474 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:50.860763 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 19:27:50.860923 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":319,"threads_started":1}
I20260926 19:27:50.862102 9245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:51.004316 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9115
I20260926 19:27:51.029420 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 8733
I20260926 19:27:51.030591 8805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:51.145071 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 8733
2026-09-26T19:27:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5325 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:51 dist-test-slave-9wb0 krb5kdc[9368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:51 dist-test-slave-9wb0 krb5kdc[9368](info): set up 2 sockets
Sep 26 19:27:51 dist-test-slave-9wb0 krb5kdc[9368](info): commencing operation
krb5kdc: starting...
W20260926 19:27:52.246431 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:52 dist-test-slave-9wb0 krb5kdc[9368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:52.470237 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:43169
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:52.794510 9381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:52.794802 9381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:52.794981 9381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:52.802883 9381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:52.803042 9381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:52.803144 9381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:52.803258 9381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:52.803355 9381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:52.813951 9381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:43169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:52.815202 9381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:52.816500 9381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:52.827983 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:52.827937 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:52.828075 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:52.828688 9381 server_base.cc:1034] running on GCE node
I20260926 19:27:52.829486 9381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:52.830817 9381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:52.832072 9381 hybrid_clock.cc:648] HybridClock initialized: now 1790450872832035 us; error 43 us; skew 500 ppm
I20260926 19:27:52.832408 9381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T19:27:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1760 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:27:53 dist-test-slave-9wb0 krb5kdc[9395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:27:53 dist-test-slave-9wb0 krb5kdc[9395](info): set up 2 sockets
Sep 26 19:27:53 dist-test-slave-9wb0 krb5kdc[9395](info): commencing operation
krb5kdc: starting...
W20260926 19:27:54.010257 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:27:54 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:27:54.244930 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41881
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:41881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:54.565187 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:54.565629 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:54.565788 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:54.573071 9408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:54.573227 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:54.573360 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:54.573470 9408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:54.573547 9408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:54.583318 9408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:41881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:54.584714 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:54.586162 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:54.598165 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:54.598210 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:54.598182 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:54.599190 9408 server_base.cc:1034] running on GCE node
I20260926 19:27:54.599961 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:54.601475 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:54.602712 9408 hybrid_clock.cc:648] HybridClock initialized: now 1790450874602663 us; error 37 us; skew 500 ppm
I20260926 19:27:54.603057 9408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:54 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:54.606165 9408 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:54.607684 9408 webserver.cc:460] Webserver started at http://127.1.39.62:46699/ using document root <none> and password file <none>
I20260926 19:27:54.608381 9408 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:54.608527 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:54.608845 9408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:54.611793 9408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "9f3f0ae7a3f34226ab48b21e0165096b"
format_stamp: "Formatted at 2026-09-26 19:27:54 on dist-test-slave-9wb0"
server_key: "4ddc1d216795fab5a3b919118a384819"
server_key_iv: "f92a63c3883440fecf4e276d7ad92287"
server_key_version: "encryptionkey@0"
I20260926 19:27:54.612577 9408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "9f3f0ae7a3f34226ab48b21e0165096b"
format_stamp: "Formatted at 2026-09-26 19:27:54 on dist-test-slave-9wb0"
server_key: "4ddc1d216795fab5a3b919118a384819"
server_key_iv: "f92a63c3883440fecf4e276d7ad92287"
server_key_version: "encryptionkey@0"
I20260926 19:27:54.617628 9408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:27:54.620900 9423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:54.622249 9408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:27:54.622423 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "9f3f0ae7a3f34226ab48b21e0165096b"
format_stamp: "Formatted at 2026-09-26 19:27:54 on dist-test-slave-9wb0"
server_key: "4ddc1d216795fab5a3b919118a384819"
server_key_iv: "f92a63c3883440fecf4e276d7ad92287"
server_key_version: "encryptionkey@0"
I20260926 19:27:54.622607 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:54.635283 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:54.639183 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:54.639472 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:54.657852 9408 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:41881
I20260926 19:27:54.657876 9474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:41881 every 8 connection(s)
I20260926 19:27:54.659224 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:54.663026 9475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:27:54.665714 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9408
I20260926 19:27:54.665937 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:27:54.666774 1180 external_mini_cluster.cc:1428] Setting key 67f6370b4dbfd09f8993333ba0126233
I20260926 19:27:54.675117 9475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Bootstrap starting.
I20260926 19:27:54.678112 9475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Neither blocks nor log segments found. Creating new log.
I20260926 19:27:54.679141 9475 log.cc:824] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:27:54 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:54.682282 9475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: No bootstrap required, opened a new log
I20260926 19:27:54.689574 9478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:54.668944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.1.39.62:41881)
0926 19:27:54.669821 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:54.669833 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:54.669879 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:27:54.671825 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:54.671849 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:54.672271 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:54.672707 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:54.674207 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:54.675554 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:54.676775 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:54.677469 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:54.677661 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:54.682056 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:54.682063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:54.682077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:54.682114 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:54.685234 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:54.685950 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:54.685965 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:54.685973 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:54.686137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:54.686694 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:54.686701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:54.686710 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:54.687280 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:54.687612 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:54.688073 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 19:27:54.688474 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":449,"threads_started":1}
I20260926 19:27:54.690440 9475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:54.690816 9475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:27:54.690936 9475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f0ae7a3f34226ab48b21e0165096b, State: Initialized, Role: FOLLOWER
I20260926 19:27:54.691609 9475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:54.691893 9475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:54.692030 9475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:54.692215 9475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:27:54.694787 9475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:54.695253 9475 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3f0ae7a3f34226ab48b21e0165096b; no voters:
I20260926 19:27:54.695775 9475 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:27:54.695926 9480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:27:54.696574 9480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 LEADER]: Becoming Leader. State: Replica: 9f3f0ae7a3f34226ab48b21e0165096b, State: Running, Role: LEADER
I20260926 19:27:54.697055 9480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:54.697840 9475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:54.699806 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } } }
I20260926 19:27:54.700181 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:54.700423 9482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3f0ae7a3f34226ab48b21e0165096b. Latest consensus state: current_term: 1 leader_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } } }
I20260926 19:27:54.700703 9482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:54.709926 9487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:54.715505 9487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:54.724745 9487 catalog_manager.cc:1347] Generated new cluster ID: 0df690bd0b5547ac99caf2c360db3682
I20260926 19:27:54.724864 9487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:54.742326 9487 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:27:54.743952 9487 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:54.752664 9487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Generated new TSK 0
I20260926 19:27:54.753650 9487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:27:54.813634 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:55.147071 9501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:55.147347 9501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:55.147437 9501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:55.154686 9501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:55.154791 9501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:55.154901 9501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:55.164597 9501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:55.165925 9501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:55.167163 9501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:55.179955 9506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:55.180019 9507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:55.180056 9509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:55.180796 9501 server_base.cc:1034] running on GCE node
I20260926 19:27:55.181496 9501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:55.182502 9501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:55.183720 9501 hybrid_clock.cc:648] HybridClock initialized: now 1790450875183648 us; error 54 us; skew 500 ppm
I20260926 19:27:55.184103 9501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:55 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450875, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:55.187655 9501 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:27:55.189105 9501 webserver.cc:460] Webserver started at http://127.1.39.1:37177/ using document root <none> and password file <none>
I20260926 19:27:55.189837 9501 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:55.189981 9501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:55.190382 9501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:55.193117 9501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "e676b6904fb2425fa1b42795c930e487"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "7fc86df8fd0e1a22bce4d7f57c9b7991"
server_key_iv: "f02c49a400c214ed03ca869ad009621f"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.193887 9501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "e676b6904fb2425fa1b42795c930e487"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "7fc86df8fd0e1a22bce4d7f57c9b7991"
server_key_iv: "f02c49a400c214ed03ca869ad009621f"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.198782 9501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 19:27:55.202185 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.203424 9501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:55.203605 9501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "e676b6904fb2425fa1b42795c930e487"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "7fc86df8fd0e1a22bce4d7f57c9b7991"
server_key_iv: "f02c49a400c214ed03ca869ad009621f"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.203805 9501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:55.227837 9501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:55.231979 9501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:55.232292 9501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:55.233254 9501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:55.234613 9501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:55.234717 9501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.234861 9501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:55.234933 9501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.268734 9501 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:43321
I20260926 19:27:55.268781 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:43321 every 8 connection(s)
I20260926 19:27:55.270097 9501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:55.276607 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9501
I20260926 19:27:55.276798 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:27:55.277508 1180 external_mini_cluster.cc:1428] Setting key 55e247d2d724300896cefddf56b153bb
Sep 26 19:27:55 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450875, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:55.291451 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:55.272624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:55599 (local address 127.1.39.62:41881)
0926 19:27:55.273043 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:55.273047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:55.273854 (+ 807us) server_negotiation.cc:407] Connection header received
0926 19:27:55.275624 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:55.275630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:55.275717 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:55.275834 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:55.277384 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.278131 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.280099 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.280848 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.280994 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:55.284564 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:55.284573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:55.284580 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:55.284623 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:55.287988 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:55.289045 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:55.289049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:55.289051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:55.289162 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:55.289715 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:55.289718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:55.289720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:55.289964 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:55.290076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:55.291047 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 19:27:55.291207 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":229,"threads_started":1}
I20260926 19:27:55.293187 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:55.272996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:55599)
0926 19:27:55.273707 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:55.273742 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:55.275359 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:55.276050 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:55.276059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:55.276531 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:55.277203 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:55.277221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.278308 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.278312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:55.279953 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:55.279965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.281047 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.281051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:55.281243 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:55.281979 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:55.282004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:55.284339 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:55.288146 (+ 3807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:55.288157 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:55.288177 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:55.288889 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:55.289272 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:55.289281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:55.289288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:55.289582 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:55.290103 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:55.290114 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:55.290598 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 19:27:55.290944 (+ 346us) client_negotiation.cc:240] Negotiation successful
0926 19:27:55.291591 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":300,"threads_started":1}
I20260926 19:27:55.294906 9629 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41881
I20260926 19:27:55.295250 9629 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:55.296008 9629 heartbeater.cc:511] Master 127.1.39.62:41881 requested a full tablet report, sending...
I20260926 19:27:55.298231 9440 ts_manager.cc:194] Registered new tserver with Master: e676b6904fb2425fa1b42795c930e487 (127.1.39.1:43321)
I20260926 19:27:55.299806 9440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:55599
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:27:55.333036 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:55.661067 9635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:55.661490 9635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:55.661630 9635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:55.673216 9635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:55.673377 9635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:55.673563 9635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:55.685904 9635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:55.687064 9635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:55.688301 9635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:55.701256 9641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:55.701437 9640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:55.701727 9643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:55.702669 9635 server_base.cc:1034] running on GCE node
I20260926 19:27:55.703251 9635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:55.704150 9635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:55.705420 9635 hybrid_clock.cc:648] HybridClock initialized: now 1790450875705345 us; error 82 us; skew 500 ppm
I20260926 19:27:55.705762 9635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:55 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450875, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:55.709508 9635 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:27:55.711344 9635 webserver.cc:460] Webserver started at http://127.1.39.2:43491/ using document root <none> and password file <none>
I20260926 19:27:55.712182 9635 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:55.712327 9635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:55.712678 9635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:55.715823 9635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "063990b25fc94e92912ca75d53c323ed"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "ec924639f556ca56b78a04884e1a52e9"
server_key_iv: "29614d0fadd2554705d8397f884c0442"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.716679 9635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "063990b25fc94e92912ca75d53c323ed"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "ec924639f556ca56b78a04884e1a52e9"
server_key_iv: "29614d0fadd2554705d8397f884c0442"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.721946 9635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:27:55.725239 9650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.726642 9635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:27:55.726794 9635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "063990b25fc94e92912ca75d53c323ed"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "ec924639f556ca56b78a04884e1a52e9"
server_key_iv: "29614d0fadd2554705d8397f884c0442"
server_key_version: "encryptionkey@0"
I20260926 19:27:55.726981 9635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:55.738116 9635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:55.741891 9635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:55.742170 9635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:55.743052 9635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:55.744349 9635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:55.744421 9635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.744583 9635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:55.744644 9635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:55.777549 9635 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:39653
I20260926 19:27:55.777637 9762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:39653 every 8 connection(s)
I20260926 19:27:55.778813 9635 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:55.780148 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9635
I20260926 19:27:55.780323 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:27:55.780893 1180 external_mini_cluster.cc:1428] Setting key c6b86c13df7ce07c9da02ea2643078c3
Sep 26 19:27:55 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450875, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:55.800823 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:55.781827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:53027 (local address 127.1.39.62:41881)
0926 19:27:55.782016 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:55.782022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:55.783316 (+ 1294us) server_negotiation.cc:407] Connection header received
0926 19:27:55.785082 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:55.785088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:55.785168 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:55.785282 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:55.786719 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.787405 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.788936 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.789435 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.789606 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:55.793380 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:55.793389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:55.793396 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:55.793441 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:55.797476 (+ 4035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:55.798281 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:55.798289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:55.798295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:55.798472 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:55.798888 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:55.798895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:55.798901 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:55.799308 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:55.799467 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:55.800135 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 19:27:55.800385 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:27:55.801709 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:55.782352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:53027)
0926 19:27:55.783175 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:55.783217 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:55.784818 (+ 1601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:55.785523 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:55.785530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:55.786010 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:55.786552 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:55.786569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.787558 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.787570 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:55.788790 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:55.788805 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:55.789683 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:55.789688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:55.789951 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:55.790694 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:55.790717 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:55.793111 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:55.797635 (+ 4524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:55.797642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:55.797656 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:55.798104 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:55.798602 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:55.798606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:55.798608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:55.798782 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:55.799450 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:55.799456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:55.799777 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 19:27:55.800016 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 19:27:55.800497 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":353,"threads_started":1}
I20260926 19:27:55.803242 9763 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41881
I20260926 19:27:55.803591 9763 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:55.804502 9763 heartbeater.cc:511] Master 127.1.39.62:41881 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:27:55.806262 9440 ts_manager.cc:194] Registered new tserver with Master: 063990b25fc94e92912ca75d53c323ed (127.1.39.2:39653)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
I20260926 19:27:55.807442 9440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:53027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:27:55.836973 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:56.161927 9768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:56.162209 9768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:56.162303 9768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:56.169536 9768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:56.169664 9768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:56.169780 9768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:56.180018 9768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:56.181290 9768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:56.182569 9768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:56.204473 9776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:56.204656 9774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:56.204782 9773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:56.205740 9768 server_base.cc:1034] running on GCE node
I20260926 19:27:56.206501 9768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:56.207590 9768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:56.208860 9768 hybrid_clock.cc:648] HybridClock initialized: now 1790450876208803 us; error 64 us; skew 500 ppm
I20260926 19:27:56.209237 9768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:56 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450876, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:56.213506 9768 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:27:56.215142 9768 webserver.cc:460] Webserver started at http://127.1.39.3:42495/ using document root <none> and password file <none>
I20260926 19:27:56.215906 9768 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:56.216051 9768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:56.216406 9768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:27:56.219975 9768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "0f1f1a298f75437188409a1a99999c0f"
format_stamp: "Formatted at 2026-09-26 19:27:56 on dist-test-slave-9wb0"
server_key: "77127c5a4f2827630182e55cbca0e510"
server_key_iv: "bd6d930a4ba2be667394f384263ee05a"
server_key_version: "encryptionkey@0"
I20260926 19:27:56.220970 9768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "0f1f1a298f75437188409a1a99999c0f"
format_stamp: "Formatted at 2026-09-26 19:27:56 on dist-test-slave-9wb0"
server_key: "77127c5a4f2827630182e55cbca0e510"
server_key_iv: "bd6d930a4ba2be667394f384263ee05a"
server_key_version: "encryptionkey@0"
I20260926 19:27:56.226951 9768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:27:56.231225 9783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:56.232815 9768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:56.233026 9768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "0f1f1a298f75437188409a1a99999c0f"
format_stamp: "Formatted at 2026-09-26 19:27:56 on dist-test-slave-9wb0"
server_key: "77127c5a4f2827630182e55cbca0e510"
server_key_iv: "bd6d930a4ba2be667394f384263ee05a"
server_key_version: "encryptionkey@0"
I20260926 19:27:56.233266 9768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:56.250844 9768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:56.255687 9768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:56.255960 9768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:56.257004 9768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:56.258422 9768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:56.258512 9768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:56.258602 9768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:56.258654 9768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:56.295974 9768 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37617
I20260926 19:27:56.296018 9895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37617 every 8 connection(s)
I20260926 19:27:56.297640 9768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:56.300241 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9768
I20260926 19:27:56.300438 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:27:56.300913 1180 external_mini_cluster.cc:1428] Setting key 5d38567065020d492ba8cf76968acf3a
I20260926 19:27:56.302932 9629 heartbeater.cc:503] Master 127.1.39.62:41881 was elected leader, sending a full tablet report...
Sep 26 19:27:56 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450876, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:27:56.319289 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:56.300367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:43667 (local address 127.1.39.62:41881)
0926 19:27:56.300618 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:56.300625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:56.302015 (+ 1390us) server_negotiation.cc:407] Connection header received
0926 19:27:56.304417 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:56.304423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:56.304502 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:56.304602 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:56.306500 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.307423 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.309230 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.309672 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.309771 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:56.313591 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:56.313596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:56.313600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:56.313629 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:56.316470 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.317231 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.317235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.317237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.317410 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.317820 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.317824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.317826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.318071 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:56.318186 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:56.318887 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 19:27:56.319042 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:27:56.320299 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:56.300901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:43667)
0926 19:27:56.301820 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:56.301879 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:56.304072 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:56.304928 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:56.304937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:56.305586 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:56.306209 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:56.306228 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.307617 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.307621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:56.309076 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:56.309090 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.309904 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.309910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:56.310182 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:56.310976 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:56.311013 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:56.313381 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:27:56.316608 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:56.316618 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:56.316636 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:56.317113 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:56.317515 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:27:56.317518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:27:56.317521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:27:56.317711 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:27:56.318183 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:27:56.318189 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:27:56.318541 (+ 352us) client_negotiation.cc:769] Sending connection context
0926 19:27:56.318780 (+ 239us) client_negotiation.cc:240] Negotiation successful
0926 19:27:56.319170 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":353,"threads_started":1}
I20260926 19:27:56.321815 9896 heartbeater.cc:347] Connected to a master server at 127.1.39.62:41881
I20260926 19:27:56.322113 9896 heartbeater.cc:464] Registering TS with master...
I20260926 19:27:56.322836 9896 heartbeater.cc:511] Master 127.1.39.62:41881 requested a full tablet report, sending...
I20260926 19:27:56.324268 9439 ts_manager.cc:194] Registered new tserver with Master: 0f1f1a298f75437188409a1a99999c0f (127.1.39.3:37617)
I20260926 19:27:56.325146 9439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:43667
I20260926 19:27:56.328949 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
Entry for principal oryx/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab.
WARNING: no policy specified for oryx/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
Entry for principal oryx/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab.
WARNING: no policy specified for oryx/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
Entry for principal oryx/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab.
I20260926 19:27:56.477034 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9501
Sep 26 19:27:56 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:27:56.499228 9906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:56.478355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.1.39.1:43321)
0926 19:27:56.481783 (+ 3428us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:56.481791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:56.481819 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:27:56.481889 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:56.481896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:56.482128 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:56.482286 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:56.483796 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.485032 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.486431 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.487011 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.487151 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:56.492739 (+ 5588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:56.492753 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:56.492774 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:56.492831 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:56.496371 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.497079 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.497086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.497088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.497214 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.497611 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.497614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.497616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.497937 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:56.498157 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:56.498618 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 19:27:56.498884 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3282,"thread_start_us":3120,"threads_started":1}
I20260926 19:27:56.500447 9624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:56.583940 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9501
I20260926 19:27:56.600421 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9635
Sep 26 19:27:56 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:27:56.615844 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:56.601382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.1.39.2:39653)
0926 19:27:56.601891 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:56.601899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:56.601922 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:27:56.601979 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:56.601985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:56.602192 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:56.602309 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:56.603466 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.604490 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.605847 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.606385 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.606486 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:56.610498 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:56.610533 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:56.610548 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:56.610581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:56.613358 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.613972 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.613979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.613981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.614092 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.614378 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.614381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.614383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.614773 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:56.614991 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:56.615246 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 19:27:56.615456 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":260,"threads_started":1}
I20260926 19:27:56.617117 9758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:56.698757 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9635
I20260926 19:27:56.715236 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9768
Sep 26 19:27:56 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:27:56.732390 9910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:56.716216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.1.39.3:37617)
0926 19:27:56.716964 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:56.716972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:56.717002 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:27:56.717058 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:56.717067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:56.717278 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:56.717516 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:56.718930 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.719921 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.721171 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:56.721793 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:56.721926 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:56.726204 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:56.726216 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:56.726231 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:56.726286 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:56.729547 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.730275 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.730292 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.730295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.730441 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:56.730828 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:56.730834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:56.730837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:56.731183 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:56.731492 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:56.731787 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 19:27:56.732043 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":348,"threads_started":1}
I20260926 19:27:56.733669 9891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:56.815977 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9768
I20260926 19:27:56.832059 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9408
I20260926 19:27:56.833168 9470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:27:56.934053 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9408
I20260926 19:27:56.955575 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41881
--webserver_interface=127.1.39.62
--webserver_port=46699
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:41881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:27:57.273669 9913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:57.273957 9913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:57.274119 9913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:57.281554 9913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:27:57.281665 9913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:57.281761 9913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:57.281863 9913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:27:57.281941 9913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:27:57.291767 9913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:41881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=46699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:57.293136 9913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:57.294463 9913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:57.306255 9919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:57.306207 9921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:57.306452 9918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:57.307540 9913 server_base.cc:1034] running on GCE node
I20260926 19:27:57.308266 9913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:57.309713 9913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:57.310941 9913 hybrid_clock.cc:648] HybridClock initialized: now 1790450877310901 us; error 50 us; skew 500 ppm
I20260926 19:27:57.311280 9913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450877, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:57.314251 9913 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:27:57.315675 9913 webserver.cc:460] Webserver started at http://127.1.39.62:46699/ using document root <none> and password file <none>
I20260926 19:27:57.316334 9913 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:57.316490 9913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:57.321703 9913 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:27:57.324337 9928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:57.325636 9913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:27:57.325814 9913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "9f3f0ae7a3f34226ab48b21e0165096b"
format_stamp: "Formatted at 2026-09-26 19:27:54 on dist-test-slave-9wb0"
server_key: "4ddc1d216795fab5a3b919118a384819"
server_key_iv: "f92a63c3883440fecf4e276d7ad92287"
server_key_version: "encryptionkey@0"
I20260926 19:27:57.326740 9913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:57.350186 9913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:57.354691 9913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:57.354959 9913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:57.373466 9913 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:41881
I20260926 19:27:57.373621 9979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:41881 every 8 connection(s)
I20260926 19:27:57.375003 9913 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:27:57.377873 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9913
I20260926 19:27:57.378638 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:43321
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:41881
--webserver_port=37177
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:27:57.382560 9980 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:27:57.386173 9980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Bootstrap starting.
I20260926 19:27:57.414783 9980 log.cc:824] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:27:57.426442 9980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:27:57.427137 9980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Bootstrap complete.
I20260926 19:27:57.441442 9980 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:57.442643 9980 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3f0ae7a3f34226ab48b21e0165096b, State: Initialized, Role: FOLLOWER
I20260926 19:27:57.443356 9980 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:57.443627 9980 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:27:57.443749 9980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:27:57.443921 9980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:27:57.448909 9980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:57.449800 9980 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3f0ae7a3f34226ab48b21e0165096b; no voters:
I20260926 19:27:57.450608 9980 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:27:57.450832 9984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:27:57.452067 9984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [term 2 LEADER]: Becoming Leader. State: Replica: 9f3f0ae7a3f34226ab48b21e0165096b, State: Running, Role: LEADER
I20260926 19:27:57.452947 9980 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:27:57.452803 9984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } }
I20260926 19:27:57.456128 9986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3f0ae7a3f34226ab48b21e0165096b. Latest consensus state: current_term: 2 leader_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } } }
I20260926 19:27:57.456517 9986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:57.466391 9985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3f0ae7a3f34226ab48b21e0165096b" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41881 } } }
I20260926 19:27:57.466807 9985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b [sys.catalog]: This master's current role is: LEADER
I20260926 19:27:57.467191 9991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:27:57.474308 9991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:27:57.476114 9991 catalog_manager.cc:1259] Loaded cluster ID: 0df690bd0b5547ac99caf2c360db3682
I20260926 19:27:57.476414 9991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:27:57.481653 9991 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:27:57.483727 9991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f3f0ae7a3f34226ab48b21e0165096b: Loaded TSK: 0
I20260926 19:27:57.485738 9991 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:27:57.782924 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:57.783239 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:57.783406 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:57.790539 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:57.790699 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:57.790871 9982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:27:57.800482 9982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.1.keytab
--principal=oryx/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:43321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=37177
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:57.801815 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:57.803058 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:57.817876 10007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:57.818013 10009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:57.817973 10006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:57.818579 9982 server_base.cc:1034] running on GCE node
I20260926 19:27:57.819324 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:57.820477 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:57.821734 9982 hybrid_clock.cc:648] HybridClock initialized: now 1790450877821659 us; error 57 us; skew 500 ppm
I20260926 19:27:57.822117 9982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450877, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:57.826257 9982 init.cc:377] Logged in from keytab as oryx/127.1.39.1@KRBTEST.COM (short username oryx)
I20260926 19:27:57.827821 9982 webserver.cc:460] Webserver started at http://127.1.39.1:37177/ using document root <none> and password file <none>
I20260926 19:27:57.828586 9982 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:57.828742 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:57.834504 9982 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:27:57.837507 10016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:57.838928 9982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:27:57.839098 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "e676b6904fb2425fa1b42795c930e487"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "7fc86df8fd0e1a22bce4d7f57c9b7991"
server_key_iv: "f02c49a400c214ed03ca869ad009621f"
server_key_version: "encryptionkey@0"
I20260926 19:27:57.840031 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:57.864468 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:57.868947 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:57.869227 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:57.870349 9982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:57.871693 9982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:57.871785 9982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:57.871922 9982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:57.872005 9982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:57.905871 9982 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:43321
I20260926 19:27:57.905927 10128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:43321 every 8 connection(s)
I20260926 19:27:57.907315 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:27:57.917472 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 9982
I20260926 19:27:57.918119 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:39653
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:41881
--webserver_port=43491
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:57.926851 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:57.910483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:37937)
0926 19:27:57.911266 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:57.911303 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:57.912972 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:57.914725 (+ 1753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:57.914733 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:57.915232 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:57.915750 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.915767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.917738 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.917742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.919056 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.919068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.920495 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.920501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.920686 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.921396 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:57.921422 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:57.925587 (+ 4165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":348,"threads_started":1}
W20260926 19:27:57.926873 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:57.910429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:37937 (local address 127.1.39.62:41881)
0926 19:27:57.913663 (+ 3234us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:57.913673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:57.913712 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:27:57.913984 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:57.913999 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:57.914302 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:57.914587 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:57.915931 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.917566 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.919196 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.920272 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.920458 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.925926 (+ 5468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:37937: BlockingRecv error: recv got EOF from 127.1.39.1:37937 (error 108)
Metrics: {"server-negotiator.queue_time_us":3038,"thread_start_us":2790,"threads_started":1}
W20260926 19:27:57.927650 10129 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:41881 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.39.62:41881: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:57.936877 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:57.928316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:41489)
0926 19:27:57.928593 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:57.928621 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:57.928709 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:57.929037 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:57.929041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:57.929440 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:57.929582 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.929591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.931119 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.931123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.932125 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.932136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.933124 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.933127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.933238 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.933867 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:57.933883 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:57.936686 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165}
W20260926 19:27:57.937233 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:57.928499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:41489 (local address 127.1.39.62:41881)
0926 19:27:57.928683 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:57.928691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:57.928712 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:57.928816 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:57.928823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:57.928902 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:57.929007 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:57.929744 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.930967 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.932261 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.932991 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.933141 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.936961 (+ 3820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:41489: BlockingRecv error: recv got EOF from 127.1.39.1:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:57 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:57.946156 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:57.937648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:49703)
0926 19:27:57.937970 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:57.937984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:57.938062 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:57.938407 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:57.938410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:57.938685 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:57.938821 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.938828 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.940374 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.940378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.941273 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:57.941283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.942348 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.942351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:57.942470 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.943095 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:57.943111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:57.945914 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232}
W20260926 19:27:57.946398 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:57.937827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:49703 (local address 127.1.39.62:41881)
0926 19:27:57.938010 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:57.938019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:57.938040 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:57.938171 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:57.938180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:57.938274 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:57.938386 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:57.938953 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.940205 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.941460 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:57.942210 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:57.942356 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:57.946150 (+ 3794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:49703: BlockingRecv error: recv got EOF from 127.1.39.1:49703 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 19:27:57.946604 10129 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:27:58.244935 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:58.245204 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:58.245384 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:58.252624 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:58.252731 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:58.252929 10133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:27:58.262609 10133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.2.keytab
--principal=oryx/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:39653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=43491
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:58.263892 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:58.265168 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:58.277911 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:58.277878 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:58.277952 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:58.278785 10133 server_base.cc:1034] running on GCE node
I20260926 19:27:58.279359 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:58.280210 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:58.281395 10133 hybrid_clock.cc:648] HybridClock initialized: now 1790450878281345 us; error 44 us; skew 500 ppm
I20260926 19:27:58.281716 10133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:58.284811 10133 init.cc:377] Logged in from keytab as oryx/127.1.39.2@KRBTEST.COM (short username oryx)
I20260926 19:27:58.286296 10133 webserver.cc:460] Webserver started at http://127.1.39.2:43491/ using document root <none> and password file <none>
I20260926 19:27:58.286921 10133 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:58.287030 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:58.292217 10133 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 19:27:58.294961 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.296253 10133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:27:58.296438 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "063990b25fc94e92912ca75d53c323ed"
format_stamp: "Formatted at 2026-09-26 19:27:55 on dist-test-slave-9wb0"
server_key: "ec924639f556ca56b78a04884e1a52e9"
server_key_iv: "29614d0fadd2554705d8397f884c0442"
server_key_version: "encryptionkey@0"
I20260926 19:27:58.297406 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:58.328042 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:58.332492 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:58.332724 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:58.333647 10133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:58.334869 10133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:58.334954 10133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.335026 10133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:58.335083 10133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.369765 10133 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:39653
I20260926 19:27:58.369958 10260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:39653 every 8 connection(s)
I20260926 19:27:58.371243 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:27:58.374084 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10133
I20260926 19:27:58.374771 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:37617
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:41881
--webserver_port=42495
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:27:58.389271 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.373716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:33637 (local address 127.1.39.62:41881)
0926 19:27:58.373927 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.373935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.375321 (+ 1386us) server_negotiation.cc:407] Connection header received
0926 19:27:58.378233 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.378240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.378315 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.378420 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.380511 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.381625 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.383652 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.384091 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.384230 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.389031 (+ 4801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:33637: BlockingRecv error: recv got EOF from 127.1.39.2:33637 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:27:58.389997 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.374278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:33637)
0926 19:27:58.375152 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.375202 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.377926 (+ 2724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.378674 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.378687 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.379529 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.380319 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.380347 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.381802 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.381810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.383496 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.383508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.384315 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.384323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.384546 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.385219 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.385239 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.388871 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":317,"threads_started":1}
W20260926 19:27:58.390650 10261 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:41881 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.39.62:41881: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:58.399518 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.391235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:49859)
0926 19:27:58.391403 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.391420 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.391495 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.391829 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.391832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.392107 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.392222 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.392229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.393792 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.393796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.394697 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.394708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.395788 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.395792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.395912 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.396516 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.396532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.399302 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 19:27:58.399852 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.391328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:49859 (local address 127.1.39.62:41881)
0926 19:27:58.391534 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.391542 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.391561 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:27:58.391613 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.391620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.391699 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.391799 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.392355 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.393628 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.394842 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.395626 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.395771 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.399576 (+ 3805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:49859: BlockingRecv error: recv got EOF from 127.1.39.2:49859 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:58.408557 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.400321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:60395)
0926 19:27:58.400502 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.400520 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.400629 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.401014 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.401018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.401267 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.401514 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.401523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.402619 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.402625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.404060 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.404071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.404710 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.404716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.404890 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.405496 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.405513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.408219 (+ 2706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:27:58.408602 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.400384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:60395 (local address 127.1.39.62:41881)
0926 19:27:58.400527 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.400534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.400570 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:27:58.400793 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.400800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.400890 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.400992 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.401696 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.402495 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.404191 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.404577 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.404702 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.408404 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:60395: BlockingRecv error: recv got EOF from 127.1.39.2:60395 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 19:27:58.408838 10261 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:27:58.719930 10264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:27:58.720247 10264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:27:58.720420 10264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:27:58.727393 10264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:27:58.727551 10264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:27:58.727742 10264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:27:58.737057 10264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/krb5kdc/oryx_127.1.39.3.keytab
--principal=oryx/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:37617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=42495
--tserver_master_addrs=127.1.39.62:41881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:27:58.738386 10264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:27:58.739682 10264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:27:58.752482 10270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:58.752671 10269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:27:58.752638 10272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:27:58.753942 10264 server_base.cc:1034] running on GCE node
I20260926 19:27:58.754803 10264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:27:58.755726 10264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:27:58.756954 10264 hybrid_clock.cc:648] HybridClock initialized: now 1790450878756893 us; error 46 us; skew 500 ppm
I20260926 19:27:58.757241 10264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450878, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:27:58.760721 10264 init.cc:377] Logged in from keytab as oryx/127.1.39.3@KRBTEST.COM (short username oryx)
I20260926 19:27:58.762259 10264 webserver.cc:460] Webserver started at http://127.1.39.3:42495/ using document root <none> and password file <none>
I20260926 19:27:58.763007 10264 fs_manager.cc:359] Metadata directory not provided
I20260926 19:27:58.763155 10264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:27:58.768524 10264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:27:58.771234 10279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.772549 10264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:27:58.772718 10264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "0f1f1a298f75437188409a1a99999c0f"
format_stamp: "Formatted at 2026-09-26 19:27:56 on dist-test-slave-9wb0"
server_key: "77127c5a4f2827630182e55cbca0e510"
server_key_iv: "bd6d930a4ba2be667394f384263ee05a"
server_key_version: "encryptionkey@0"
I20260926 19:27:58.773660 10264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:27:58.791632 10264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:27:58.796025 10264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:27:58.796303 10264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:27:58.797281 10264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:27:58.798643 10264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:27:58.798750 10264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.798870 10264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:27:58.798961 10264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:27:58.832408 10264 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37617
I20260926 19:27:58.832507 10391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37617 every 8 connection(s)
I20260926 19:27:58.833900 10264 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:27:58.840804 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10264
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 19:27:58.851325 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.836412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:49355 (local address 127.1.39.62:41881)
0926 19:27:58.836612 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.836619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.837844 (+ 1225us) server_negotiation.cc:407] Connection header received
0926 19:27:58.839595 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.839599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.839679 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.839776 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.841697 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.842762 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.844819 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.845347 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.845448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.851063 (+ 5615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:49355: BlockingRecv error: recv got EOF from 127.1.39.3:49355 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:27:58.852084 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.836936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:49355)
0926 19:27:58.837693 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.837729 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.839345 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.840067 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.840077 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.840645 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.841488 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.841522 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.842962 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.842969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.844672 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.844687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.845589 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.845599 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.845943 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.846801 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.846828 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.850895 (+ 4067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":334,"threads_started":1}
W20260926 19:27:58.853117 10392 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:41881 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.39.62:41881: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
I20260926 19:27:58.859368 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.842034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.1.39.62:41881)
0926 19:27:58.842998 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.843005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.843026 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:27:58.843089 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.843095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.843174 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.843305 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.844917 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.846232 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.847631 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.848326 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.848485 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.851601 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:27:58.851611 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:27:58.851628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:27:58.851669 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:27:58.855727 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:58.856519 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:58.856525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:58.856532 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:58.856715 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:27:58.857184 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:27:58.857188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:27:58.857190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:27:58.857859 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:27:58.858166 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 19:27:58.858592 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 19:27:58.858957 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":602,"threads_started":1}
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:58.863129 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.853793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:36163)
0926 19:27:58.854213 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.854231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.854328 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.854661 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.854665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.855002 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.855152 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.855161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.856383 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.856389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.857858 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.857872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.858981 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.858987 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.859162 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.859826 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.859856 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.862835 (+ 2979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321}
W20260926 19:27:58.863770 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.854049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:36163 (local address 127.1.39.62:41881)
0926 19:27:58.854223 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.854230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.854281 (+ 51us) server_negotiation.cc:407] Connection header received
0926 19:27:58.854439 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.854444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.854520 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.854620 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.855301 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.856229 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.858039 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.858668 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.858781 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.863620 (+ 4839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:36163: BlockingRecv error: recv got EOF from 127.1.39.3:36163 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:58.871528 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.864342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:44917)
0926 19:27:58.864539 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.864558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.864653 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.864945 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.864949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.865243 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.865451 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.865459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.866429 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.866436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.867362 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.867378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.868026 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.868032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.868215 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.868765 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.868781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.871331 (+ 2550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:27:58.871820 10392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:27:58.871869 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.864431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:44917 (local address 127.1.39.62:41881)
0926 19:27:58.864640 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.864645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.864662 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:27:58.864777 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.864780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.864829 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.864923 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.865600 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.866300 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.867498 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.867889 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.868024 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.871504 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:44917: BlockingRecv error: recv got EOF from 127.1.39.3:44917 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:58 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:58.956269 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:58.947323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:33671)
0926 19:27:58.947923 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:58.947939 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:58.948124 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:58.948534 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:58.948539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:58.949146 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:58.949469 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.949484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.950473 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.950477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.951448 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:58.951459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.952175 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.952179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:58.952293 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.953025 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:58.953057 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:58.956016 (+ 2959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":340,"threads_started":1}
W20260926 19:27:58.956365 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:58.947394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:33671 (local address 127.1.39.62:41881)
0926 19:27:58.947643 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:58.947652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:58.948049 (+ 397us) server_negotiation.cc:407] Connection header received
0926 19:27:58.948268 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:58.948276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:58.948364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:58.948510 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:58.949610 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.950329 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.951623 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:58.952024 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:58.952124 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:58.956205 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:33671: BlockingRecv error: recv got EOF from 127.1.39.1:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:59.418550 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:59.409422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:56473)
0926 19:27:59.410056 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:59.410075 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:59.410272 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:59.410669 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:59.410674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:59.411204 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:59.411408 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.411419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.412475 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.412483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.413756 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.413769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.414577 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.414585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.414781 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.415443 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:59.415463 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:59.418330 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":313,"threads_started":1}
W20260926 19:27:59.418778 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:59.409613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:56473 (local address 127.1.39.62:41881)
0926 19:27:59.409847 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:59.409855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:59.410182 (+ 327us) server_negotiation.cc:407] Connection header received
0926 19:27:59.410454 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:59.410458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:59.410521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:59.410638 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:59.411563 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.412313 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.413902 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.414376 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.414540 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.418537 (+ 3997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:56473: BlockingRecv error: recv got EOF from 127.1.39.2:56473 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:59.881192 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:59.872404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:37055)
0926 19:27:59.872978 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:59.872995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:59.873103 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:59.873398 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:59.873401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:59.873845 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:59.874036 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.874045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.875046 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.875051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.876064 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.876075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.876776 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.876782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.876921 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.877602 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:59.877619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:59.880873 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":336,"threads_started":1}
W20260926 19:27:59.881315 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:59.872505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:37055 (local address 127.1.39.62:41881)
0926 19:27:59.872656 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:59.872660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:59.873065 (+ 405us) server_negotiation.cc:407] Connection header received
0926 19:27:59.873200 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:59.873204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:59.873261 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:59.873355 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:59.874166 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.874889 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.876193 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.876587 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.876682 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.881060 (+ 4378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:37055: BlockingRecv error: recv got EOF from 127.1.39.3:37055 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:27:59 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:27:59.965740 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:27:59.957193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:37451)
0926 19:27:59.957720 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 19:27:59.957737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:27:59.957911 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:27:59.958249 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:27:59.958253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:27:59.958669 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:27:59.958811 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.958820 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.959819 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.959825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.960731 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:27:59.960742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.961485 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.961491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:27:59.961653 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.962275 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:27:59.962292 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:27:59.965508 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":303,"threads_started":1}
W20260926 19:27:59.966012 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:27:59.957268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:37451 (local address 127.1.39.62:41881)
0926 19:27:59.957551 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 19:27:59.957560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:27:59.957841 (+ 281us) server_negotiation.cc:407] Connection header received
0926 19:27:59.958035 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:27:59.958043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:27:59.958113 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:27:59.958215 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:27:59.958954 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.959681 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.960868 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:27:59.961335 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:27:59.961472 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:27:59.965674 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:37451: BlockingRecv error: recv got EOF from 127.1.39.1:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:00.427841 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:00.419517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:56455)
0926 19:28:00.420029 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:00.420045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:00.420201 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:00.420575 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:00.420580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:00.420980 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:00.421116 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.421124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.422180 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.422186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.423084 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.423094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.423782 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.423787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.423949 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.424541 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:00.424557 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:00.427566 (+ 3009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
W20260926 19:28:00.427922 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:00.419545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:56455 (local address 127.1.39.62:41881)
0926 19:28:00.419775 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:00.419783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:00.420134 (+ 351us) server_negotiation.cc:407] Connection header received
0926 19:28:00.420343 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:00.420348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:00.420419 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:00.420521 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:00.421259 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.422040 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.423220 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.423635 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.423772 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.427758 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:56455: BlockingRecv error: recv got EOF from 127.1.39.2:56455 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:00.891835 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:00.882168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:55567)
0926 19:28:00.882780 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:00.882798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:00.883001 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:00.883375 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:00.883380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:00.883857 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:00.884093 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.884109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.885246 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.885252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.886627 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.886643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.887639 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.887644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.887768 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.888390 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:00.888409 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:00.891594 (+ 3185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":374,"threads_started":1}
W20260926 19:28:00.892168 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:00.882397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:55567 (local address 127.1.39.62:41881)
0926 19:28:00.882671 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:00.882679 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:00.882892 (+ 213us) server_negotiation.cc:407] Connection header received
0926 19:28:00.883155 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:00.883160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:00.883235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:00.883350 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:00.884263 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.885082 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.886799 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.887414 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.887613 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.891861 (+ 4248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:55567: BlockingRecv error: recv got EOF from 127.1.39.3:55567 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:00 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:00.976962 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:00.967199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:54623)
0926 19:28:00.968592 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:00.968609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:00.968809 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:00.969190 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:00.969197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:00.969622 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:00.969801 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.969810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.971015 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.971022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.971921 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:00.971933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.972628 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.972634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:00.972813 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.973559 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:00.973578 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:00.976601 (+ 3023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":890,"threads_started":1}
W20260926 19:28:00.977197 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:00.967401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:54623 (local address 127.1.39.62:41881)
0926 19:28:00.968053 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:00.968061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:00.968717 (+ 656us) server_negotiation.cc:407] Connection header received
0926 19:28:00.968950 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:00.968955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:00.969042 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:00.969150 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:00.969957 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.970845 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.972071 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:00.972465 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:00.972601 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:00.976934 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:54623: BlockingRecv error: recv got EOF from 127.1.39.1:54623 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:01.437788 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:01.428945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:48405)
0926 19:28:01.429536 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:01.429556 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:01.429686 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:01.430038 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:01.430041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:01.430412 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:01.430602 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.430611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.431766 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.431770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.432827 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.432839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.433587 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.433591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.433725 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.434298 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:01.434316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:01.437453 (+ 3137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":350,"threads_started":1}
W20260926 19:28:01.437951 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:01.429033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:48405 (local address 127.1.39.62:41881)
0926 19:28:01.429222 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:01.429227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:01.429633 (+ 406us) server_negotiation.cc:407] Connection header received
0926 19:28:01.429813 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:01.429817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:01.429877 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:01.429962 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:01.430726 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.431652 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.432952 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.433430 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.433540 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.437681 (+ 4141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:48405: BlockingRecv error: recv got EOF from 127.1.39.2:48405 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:01.900584 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:01.892733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:35953)
0926 19:28:01.893173 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:01.893189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:01.893335 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:01.893640 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:01.893644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:01.893958 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:01.894137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.894145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.895153 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.895157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.896064 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.896074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.896728 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.896732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.896856 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.897530 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:01.897548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:01.900342 (+ 2794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
W20260926 19:28:01.900867 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:01.892819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:35953 (local address 127.1.39.62:41881)
0926 19:28:01.893081 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:01.893087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:01.893271 (+ 184us) server_negotiation.cc:407] Connection header received
0926 19:28:01.893454 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:01.893458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:01.893518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:01.893598 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:01.894277 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.895031 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.896185 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.896612 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.896705 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.900596 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:35953: BlockingRecv error: recv got EOF from 127.1.39.3:35953 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:01 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:01.986023 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:01.977928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:57727)
0926 19:28:01.978529 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:01.978545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:01.978705 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:01.979053 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:01.979058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:01.979440 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:01.979583 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.979592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.980575 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.980583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.981614 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:01.981625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.982309 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.982317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:01.982506 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.983129 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:01.983146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:01.985806 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":335,"threads_started":1}
W20260926 19:28:01.986320 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:01.977994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:57727 (local address 127.1.39.62:41881)
0926 19:28:01.978294 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:01.978302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:01.978637 (+ 335us) server_negotiation.cc:407] Connection header received
0926 19:28:01.978829 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:01.978835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:01.978905 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:01.979021 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:01.979731 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.980442 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.981747 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:01.982165 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:01.982296 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:01.986033 (+ 3737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:57727: BlockingRecv error: recv got EOF from 127.1.39.1:57727 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:02.447134 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:02.438746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:49759)
0926 19:28:02.439265 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:02.439281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:02.439454 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:02.439800 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:02.439805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:02.440205 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:02.440350 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.440358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.441370 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.441376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.442393 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.442404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.443120 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.443126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.443308 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.443920 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:02.443937 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:02.446869 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":266,"threads_started":1}
W20260926 19:28:02.447252 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:02.438821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:49759 (local address 127.1.39.62:41881)
0926 19:28:02.439083 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:02.439089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:02.439383 (+ 294us) server_negotiation.cc:407] Connection header received
0926 19:28:02.439601 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:02.439608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:02.439673 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:02.439776 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:02.440490 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.441208 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.442549 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.442974 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.443111 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.447034 (+ 3923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:49759: BlockingRecv error: recv got EOF from 127.1.39.2:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:02.915608 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:02.901561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:46889)
0926 19:28:02.902062 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:02.902079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:02.902234 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:02.902564 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:02.902569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:02.902942 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:02.903090 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.903098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.904118 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.904124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.905067 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.905078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.905817 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.905822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.906029 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.912052 (+ 6023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:02.912070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:02.915283 (+ 3213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":263,"threads_started":1}
W20260926 19:28:02.915745 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:02.901649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:46889 (local address 127.1.39.62:41881)
0926 19:28:02.901848 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:02.901857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:02.902166 (+ 309us) server_negotiation.cc:407] Connection header received
0926 19:28:02.902363 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:02.902368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:02.902435 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:02.902542 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:02.903230 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.903980 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.905205 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.905678 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.905807 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.915498 (+ 9691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:46889: BlockingRecv error: recv got EOF from 127.1.39.3:46889 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:02 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:02.996150 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:02.986972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:51369)
0926 19:28:02.987574 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:02.987592 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:02.987762 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:02.988175 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:02.988180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:02.988659 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:02.988910 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.988931 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.990062 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.990071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.991407 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:02.991420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.992173 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.992179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:02.992439 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.993123 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:02.993140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:02.995916 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":278,"threads_started":1}
W20260926 19:28:02.996408 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:02.987117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:51369 (local address 127.1.39.62:41881)
0926 19:28:02.987371 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:02.987380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:02.987688 (+ 308us) server_negotiation.cc:407] Connection header received
0926 19:28:02.987938 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:02.987944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:02.988024 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:02.988141 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:02.989077 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.989914 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.991576 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:02.992015 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:02.992159 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:02.996131 (+ 3972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:51369: BlockingRecv error: recv got EOF from 127.1.39.1:51369 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:03.457389 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:03.448177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:49125)
0926 19:28:03.448724 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:03.448741 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:03.448855 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:03.449184 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:03.449188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:03.449625 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:03.449797 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:03.449806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.450892 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.450896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:03.451852 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:03.451864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.452819 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.452826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:03.453042 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:03.453847 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:03.453874 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:03.457083 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":279,"threads_started":1}
W20260926 19:28:03.457782 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:03.448411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:49125 (local address 127.1.39.62:41881)
0926 19:28:03.448595 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:03.448599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:03.448818 (+ 219us) server_negotiation.cc:407] Connection header received
0926 19:28:03.448963 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:03.448967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:03.449034 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:03.449125 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:03.449933 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.450751 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.452044 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.452543 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.452753 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:03.457468 (+ 4715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:49125: BlockingRecv error: recv got EOF from 127.1.39.2:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:03.925773 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:03.916747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:59495)
0926 19:28:03.917284 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:03.917319 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:03.917426 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:03.917779 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:03.917784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:03.918356 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:03.918608 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:03.918623 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.920017 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.920021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:03.920934 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:03.920945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.921735 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.921739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:03.921861 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:03.922474 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:03.922491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:03.925531 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":286,"threads_started":1}
W20260926 19:28:03.925920 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:03.916880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:59495 (local address 127.1.39.62:41881)
0926 19:28:03.917226 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:03.917231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:03.917418 (+ 187us) server_negotiation.cc:407] Connection header received
0926 19:28:03.917530 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:03.917552 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:03.917618 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:03.917717 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:03.918771 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.919879 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.921072 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.921598 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.921698 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:03.925708 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:59495: BlockingRecv error: recv got EOF from 127.1.39.3:59495 (error 108)
Metrics: {"server-negotiator.queue_time_us":227}
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:03 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:04.005491 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:03.997114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:36169)
0926 19:28:03.997684 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:03.997700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:03.997809 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:03.998144 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:03.998148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:03.998508 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:03.998662 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:03.998671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.999724 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:03.999727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.000767 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:04.000781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.001575 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.001582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.001713 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.002301 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:04.002317 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:04.005210 (+ 2893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":323,"threads_started":1}
W20260926 19:28:04.005795 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:03.997246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:36169 (local address 127.1.39.62:41881)
0926 19:28:03.997499 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:03.997504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:03.997772 (+ 268us) server_negotiation.cc:407] Connection header received
0926 19:28:03.997905 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:03.997908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:03.997982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:03.998074 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:03.998784 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:03.999605 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.000897 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.001414 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.001582 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.005468 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:36169: BlockingRecv error: recv got EOF from 127.1.39.1:36169 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 19:28:04 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:04 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:04.467036 10413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:04.458510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:45615)
0926 19:28:04.459021 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:04.459038 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:04.459155 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:04.459583 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:04.459588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:04.459981 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:04.460144 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:04.460154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.461286 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.461313 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.462288 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:04.462301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.463120 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.463126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.463300 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.463877 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:04.463894 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:04.466789 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":302,"threads_started":1}
W20260926 19:28:04.467296 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:04.458595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:45615 (local address 127.1.39.62:41881)
0926 19:28:04.458824 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:04.458834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:04.459138 (+ 304us) server_negotiation.cc:407] Connection header received
0926 19:28:04.459311 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:04.459321 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:04.459388 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:04.459474 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:04.460311 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.461129 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.462441 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.462966 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.463105 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.467045 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:45615: BlockingRecv error: recv got EOF from 127.1.39.2:45615 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 19:28:04 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:04 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:04.935595 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:04.926826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:48463)
0926 19:28:04.927373 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:04.927389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:04.927551 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:04.927875 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:04.927879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:04.928273 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:04.928409 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:04.928417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.929477 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.929484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.930504 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:04.930515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.931296 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.931302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:04.931481 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.932101 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:04.932118 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:04.935277 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":317,"threads_started":1}
W20260926 19:28:04.935758 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:04.926948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:48463 (local address 127.1.39.62:41881)
0926 19:28:04.927186 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:04.927194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:04.927481 (+ 287us) server_negotiation.cc:407] Connection header received
0926 19:28:04.927689 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:04.927693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:04.927751 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:04.927853 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:04.928555 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.929337 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.930635 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:04.931135 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:04.931290 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:04.935513 (+ 4223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:48463: BlockingRecv error: recv got EOF from 127.1.39.3:48463 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:05.015322 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:05.006583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:44705)
0926 19:28:05.007346 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:05.007370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:05.007530 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:05.007873 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:05.007876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:05.008308 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:05.008497 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.008505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.009533 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.009537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.010542 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.010554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.011211 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.011216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.011356 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.012053 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:05.012072 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:05.014952 (+ 2880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":333,"threads_started":1}
W20260926 19:28:05.015367 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:05.006607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:44705 (local address 127.1.39.62:41881)
0926 19:28:05.006821 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:05.006830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:05.007506 (+ 676us) server_negotiation.cc:407] Connection header received
0926 19:28:05.007675 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:05.007679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:05.007740 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:05.007825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:05.008662 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.009398 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.010672 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.011071 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.011170 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.015148 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:44705: BlockingRecv error: recv got EOF from 127.1.39.1:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:05.476648 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:05.468038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:57487)
0926 19:28:05.468642 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:05.468673 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:05.468804 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:05.469160 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:05.469164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:05.469688 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:05.469861 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.469871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.470910 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.470914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.471858 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.471870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.472516 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.472521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.472633 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.473208 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:05.473236 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:05.476411 (+ 3175us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":240,"threads_started":1}
W20260926 19:28:05.476922 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:05.468126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:57487 (local address 127.1.39.62:41881)
0926 19:28:05.468402 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:05.468409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:05.468776 (+ 367us) server_negotiation.cc:407] Connection header received
0926 19:28:05.468916 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:05.468920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:05.469012 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:05.469101 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:05.469995 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.470761 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.471983 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.472365 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.472455 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.476612 (+ 4157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:57487: BlockingRecv error: recv got EOF from 127.1.39.2:57487 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:05 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:05.944774 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:05.936448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:43561)
0926 19:28:05.937014 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:05.937030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:05.937211 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:05.937573 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:05.937577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:05.937994 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:05.938167 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.938175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.939227 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.939234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.940162 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:05.940173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.940922 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.940928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:05.941093 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.941698 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:05.941714 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:05.944391 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":243,"threads_started":1}
W20260926 19:28:05.944816 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:05.936560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:43561 (local address 127.1.39.62:41881)
0926 19:28:05.936761 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:05.936769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:05.937120 (+ 351us) server_negotiation.cc:407] Connection header received
0926 19:28:05.937371 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:05.937382 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:05.937443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:05.937544 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:05.938309 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.939077 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.940300 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:05.940736 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:05.940873 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:05.944592 (+ 3719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:43561: BlockingRecv error: recv got EOF from 127.1.39.3:43561 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:06.024739 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:06.016231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:48131)
0926 19:28:06.016780 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:06.016797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:06.016968 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:06.017381 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:06.017390 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:06.017965 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:06.018171 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.018185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.019254 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.019258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.020192 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.020204 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.020901 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.020905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.021014 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.021658 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:06.021675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:06.024506 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":337,"threads_started":1}
W20260926 19:28:06.024923 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:06.016444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:48131 (local address 127.1.39.62:41881)
0926 19:28:06.016689 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:06.016696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:06.016932 (+ 236us) server_negotiation.cc:407] Connection header received
0926 19:28:06.017106 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:06.017112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:06.017189 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:06.017341 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:06.018362 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.019125 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.020329 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.020754 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.020846 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.024691 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:48131: BlockingRecv error: recv got EOF from 127.1.39.1:48131 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:06.488714 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:06.477609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:40113)
0926 19:28:06.478115 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:06.478132 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:06.478289 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:06.478840 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:06.478847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:06.479335 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:06.479535 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.479547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.480838 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.480845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.481943 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.481957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.482814 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.482822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.483013 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.483785 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:06.483822 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:06.488337 (+ 4515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
W20260926 19:28:06.488816 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:06.477680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:40113 (local address 127.1.39.62:41881)
0926 19:28:06.477941 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:06.477949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:06.478215 (+ 266us) server_negotiation.cc:407] Connection header received
0926 19:28:06.478423 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:06.478428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:06.478587 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:06.478817 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:06.479709 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.480669 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.482107 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.482606 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.482776 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.488591 (+ 5815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:40113: BlockingRecv error: recv got EOF from 127.1.39.2:40113 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:06 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:06.954113 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:06.945776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:55943)
0926 19:28:06.946268 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:06.946284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:06.946401 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:06.946735 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:06.946738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:06.947124 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:06.947269 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.947277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.948318 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.948321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.949223 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:06.949233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.949987 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.949994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:06.950102 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.950704 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:06.950723 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:06.953812 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
W20260926 19:28:06.954237 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:06.945844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:55943 (local address 127.1.39.62:41881)
0926 19:28:06.946058 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:06.946062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:06.946410 (+ 348us) server_negotiation.cc:407] Connection header received
0926 19:28:06.946530 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:06.946534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:06.946607 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:06.946709 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:06.947429 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.948191 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.949408 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:06.949810 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:06.949960 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:06.953997 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:55943: BlockingRecv error: recv got EOF from 127.1.39.3:55943 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:07.035648 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:07.025798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:37969)
0926 19:28:07.026340 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:07.026358 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:07.026491 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:07.026914 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:07.026918 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:07.027293 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:07.027586 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.027607 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.029387 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.029394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.030489 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.030502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.031399 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.031404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.031554 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.032170 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:07.032196 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:07.035389 (+ 3193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
W20260926 19:28:07.035841 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:07.025995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:37969 (local address 127.1.39.62:41881)
0926 19:28:07.026275 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:07.026284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:07.026481 (+ 197us) server_negotiation.cc:407] Connection header received
0926 19:28:07.026682 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:07.026687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:07.026754 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:07.026848 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:07.027881 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.029166 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.030678 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.031194 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.031313 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.035595 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:37969: BlockingRecv error: recv got EOF from 127.1.39.1:37969 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:07.506174 10422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:07.490237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:59237)
0926 19:28:07.490832 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:07.490849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:07.491001 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:07.491414 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:07.491420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:07.491959 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:07.492184 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.492195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.493495 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.493499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.494418 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.494429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.495137 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.495142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.495261 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.502558 (+ 7297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:07.502584 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:07.505928 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":343,"threads_started":1}
W20260926 19:28:07.506341 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:07.490297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:59237 (local address 127.1.39.62:41881)
0926 19:28:07.490517 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:07.490522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:07.490949 (+ 427us) server_negotiation.cc:407] Connection header received
0926 19:28:07.491150 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:07.491158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:07.491239 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:07.491350 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:07.492357 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.493368 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.494558 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.494993 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.495085 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.506128 (+ 11043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:59237: BlockingRecv error: recv got EOF from 127.1.39.2:59237 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:07 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:07.971254 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:07.955102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:35697)
0926 19:28:07.955690 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:07.955706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:07.955864 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:07.956209 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:07.956214 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:07.956639 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:07.956810 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.956819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.957911 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.957917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.958881 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:07.958893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.959936 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.959942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:07.960148 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.968170 (+ 8022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:07.968191 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:07.971015 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":307,"threads_started":1}
W20260926 19:28:07.971628 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:07.955202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:35697 (local address 127.1.39.62:41881)
0926 19:28:07.955435 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:07.955444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:07.955795 (+ 351us) server_negotiation.cc:407] Connection header received
0926 19:28:07.956009 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:07.956013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:07.956075 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:07.956174 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:07.956980 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.957773 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.959031 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:07.959733 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:07.959921 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:07.971337 (+ 11416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:35697: BlockingRecv error: recv got EOF from 127.1.39.3:35697 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:08.044629 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:08.036576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:40969)
0926 19:28:08.037082 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:08.037099 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:08.037209 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:08.037539 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:08.037543 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:08.037878 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:08.038020 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.038029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.038972 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.038976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.039921 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.039932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.040561 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.040565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.040672 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.041327 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:08.041353 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:08.044383 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":279,"threads_started":1}
W20260926 19:28:08.044824 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:08.036675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:40969 (local address 127.1.39.62:41881)
0926 19:28:08.036874 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:08.036879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:08.037172 (+ 293us) server_negotiation.cc:407] Connection header received
0926 19:28:08.037318 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:08.037322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:08.037380 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:08.037456 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:08.038131 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.038845 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.040032 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.040431 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.040518 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.044578 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:40969: BlockingRecv error: recv got EOF from 127.1.39.1:40969 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:08.515715 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:08.507182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:51487)
0926 19:28:08.507731 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:08.507755 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:08.507866 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:08.508167 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:08.508171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:08.508512 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:08.508655 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.508663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.509750 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.509755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.510973 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.510988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.511949 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.511956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.512123 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.512684 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:08.512701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:08.515448 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":324,"threads_started":1}
W20260926 19:28:08.516019 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:08.507296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:51487 (local address 127.1.39.62:41881)
0926 19:28:08.507497 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:08.507505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:08.507849 (+ 344us) server_negotiation.cc:407] Connection header received
0926 19:28:08.507975 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:08.507979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:08.508042 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:08.508129 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:08.508784 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.509603 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.511158 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.511790 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.511927 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.515709 (+ 3782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:51487: BlockingRecv error: recv got EOF from 127.1.39.2:51487 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:08 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:08.981149 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:08.972224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:41815)
0926 19:28:08.972778 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:08.972795 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:08.972914 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:08.973274 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:08.973278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:08.973900 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:08.974105 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.974121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.975337 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.975341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.976431 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:08.976444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.977131 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.977136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:08.977260 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.978069 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:08.978098 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:08.980928 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":317,"threads_started":1}
W20260926 19:28:08.981352 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:08.972349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:41815 (local address 127.1.39.62:41881)
0926 19:28:08.972585 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:08.972589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:08.972866 (+ 277us) server_negotiation.cc:407] Connection header received
0926 19:28:08.973035 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:08.973039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:08.973116 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:08.973204 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:08.974288 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.975193 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.976590 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:08.976987 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:08.977081 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:08.981093 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:41815: BlockingRecv error: recv got EOF from 127.1.39.3:41815 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:09.053525 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:09.045584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:52583)
0926 19:28:09.046172 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:09.046187 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:09.046299 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:09.046573 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:09.046575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:09.046972 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:09.047144 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.047153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.048092 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.048096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.048969 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.048979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.049707 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.049712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.049837 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.050495 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:09.050511 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:09.053196 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":389,"threads_started":1}
W20260926 19:28:09.053653 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:09.045645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:52583 (local address 127.1.39.62:41881)
0926 19:28:09.045846 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:09.045852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:09.046268 (+ 416us) server_negotiation.cc:407] Connection header received
0926 19:28:09.046413 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:09.046417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:09.046484 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:09.046556 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:09.047267 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.047982 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.049093 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.049572 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.049660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.053419 (+ 3759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:52583: BlockingRecv error: recv got EOF from 127.1.39.1:52583 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:09.525017 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:09.516712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:34837)
0926 19:28:09.517227 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:09.517243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:09.517418 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:09.517713 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:09.517717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:09.518104 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:09.518288 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.518296 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.519320 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.519324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.520191 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.520202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.520847 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.520851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.520967 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.521575 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:09.521592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:09.524790 (+ 3198us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":296,"threads_started":1}
W20260926 19:28:09.525233 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:09.516929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:34837 (local address 127.1.39.62:41881)
0926 19:28:09.517104 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:09.517108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:09.517323 (+ 215us) server_negotiation.cc:407] Connection header received
0926 19:28:09.517525 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:09.517529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:09.517589 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:09.517669 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:09.518442 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.519193 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.520315 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.520731 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.520823 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.524988 (+ 4165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:34837: BlockingRecv error: recv got EOF from 127.1.39.2:34837 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:09 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:09.990061 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:09.982037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:57719)
0926 19:28:09.982558 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:09.982576 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:09.982702 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:09.982991 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:09.982994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:09.983398 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:09.983625 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.983634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.984717 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.984725 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.985646 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:09.985659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.986343 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.986349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:09.986524 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.987094 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:09.987111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:09.989831 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":275,"threads_started":1}
W20260926 19:28:09.990266 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:09.982092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:57719 (local address 127.1.39.62:41881)
0926 19:28:09.982312 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:09.982322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:09.982656 (+ 334us) server_negotiation.cc:407] Connection header received
0926 19:28:09.982804 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:09.982808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:09.982866 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:09.982944 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:09.983759 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.984583 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.985785 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:09.986190 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:09.986332 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:09.990044 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:57719: BlockingRecv error: recv got EOF from 127.1.39.3:57719 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:10.062844 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:10.054602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:54773)
0926 19:28:10.055099 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:10.055115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:10.055280 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:10.055717 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:10.055722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:10.056232 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:10.056416 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.056425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.057572 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.057578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.058493 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.058505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.059193 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.059197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.059316 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.059869 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:10.059886 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:10.062625 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":266,"threads_started":1}
W20260926 19:28:10.063019 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:10.054637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:54773 (local address 127.1.39.62:41881)
0926 19:28:10.054873 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:10.054880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:10.055196 (+ 316us) server_negotiation.cc:407] Connection header received
0926 19:28:10.055452 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:10.055459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:10.055540 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:10.055642 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:10.056583 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.057421 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.058607 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.059077 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.059166 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.062786 (+ 3620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:54773: BlockingRecv error: recv got EOF from 127.1.39.1:54773 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:10.533840 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:10.525901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:33623)
0926 19:28:10.526383 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:10.526399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:10.526515 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:10.526812 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:10.526815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:10.527173 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:10.527322 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.527330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.528255 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.528259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.529188 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.529199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.529903 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.529908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.530044 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.530669 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:10.530685 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:10.533492 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
W20260926 19:28:10.533881 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:10.525974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:33623 (local address 127.1.39.62:41881)
0926 19:28:10.526207 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:10.526214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:10.526473 (+ 259us) server_negotiation.cc:407] Connection header received
0926 19:28:10.526640 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:10.526643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:10.526702 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:10.526773 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:10.527432 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.528126 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.529351 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.529754 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.529844 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.533660 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:33623: BlockingRecv error: recv got EOF from 127.1.39.2:33623 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:10 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:10.998752 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:10.991029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:43569)
0926 19:28:10.991516 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:10.991533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:10.991645 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:10.991921 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:10.991924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:10.992275 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:10.992421 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.992429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.993421 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.993425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.994318 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:10.994329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.995082 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.995086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:10.995214 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.995890 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:10.995908 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:10.998530 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
W20260926 19:28:10.999029 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:10.991053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:43569 (local address 127.1.39.62:41881)
0926 19:28:10.991264 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:10.991273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:10.991610 (+ 337us) server_negotiation.cc:407] Connection header received
0926 19:28:10.991742 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:10.991746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:10.991805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:10.991881 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:10.992542 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.993276 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.994472 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:10.994953 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:10.995057 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:10.998727 (+ 3670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:43569: BlockingRecv error: recv got EOF from 127.1.39.3:43569 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 19:28:11 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:11 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:11.072672 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:11.063810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:36423)
0926 19:28:11.064350 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:11.064369 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:11.064567 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:11.064942 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:11.064947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:11.065432 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:11.065624 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:11.065634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.066662 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.066668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:11.067731 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:11.067744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.068526 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.068533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:11.068732 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:11.069377 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:11.069395 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:11.072309 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":278,"threads_started":1}
W20260926 19:28:11.072930 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:11.063884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:36423 (local address 127.1.39.62:41881)
0926 19:28:11.064129 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:11.064137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:11.064490 (+ 353us) server_negotiation.cc:407] Connection header received
0926 19:28:11.064714 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:11.064718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:11.064798 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:11.064916 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:11.065776 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.066519 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.067870 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.068334 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.068484 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:11.072545 (+ 4061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:36423: BlockingRecv error: recv got EOF from 127.1.39.1:36423 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 19:28:11 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:11 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:11.543154 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:11.534801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:48563)
0926 19:28:11.535358 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:11.535373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:11.535548 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:11.535880 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:11.535884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:11.536294 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:11.536480 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:11.536489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.537694 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.537700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:11.538740 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:11.538751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.539490 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.539495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:11.539647 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:11.540190 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:11.540207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:11.542819 (+ 2612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":304,"threads_started":1}
W20260926 19:28:11.543238 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:11.535015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:48563 (local address 127.1.39.62:41881)
0926 19:28:11.535241 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:11.535249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:11.535454 (+ 205us) server_negotiation.cc:407] Connection header received
0926 19:28:11.535683 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:11.535688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:11.535747 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:11.535846 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:11.536617 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.537563 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.538869 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:11.539313 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:11.539444 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:11.543004 (+ 3560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:48563: BlockingRecv error: recv got EOF from 127.1.39.2:48563 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:12.008296 10435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:11.999652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:37639)
0926 19:28:12.000185 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:12.000228 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:12.000338 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:12.000664 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:12.000668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:12.001250 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:12.001519 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.001529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.002463 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.002469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.003374 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.003385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.004068 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.004073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.004185 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.004828 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:12.004846 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:12.007946 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
W20260926 19:28:12.008387 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:11.999784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:37639 (local address 127.1.39.62:41881)
0926 19:28:11.999954 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:11.999959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:12.000301 (+ 342us) server_negotiation.cc:407] Connection header received
0926 19:28:12.000443 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:12.000447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:12.000508 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:12.000601 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:12.001648 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.002347 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.003508 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.003908 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.004008 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.008162 (+ 4154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:37639: BlockingRecv error: recv got EOF from 127.1.39.3:37639 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:12.082451 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:12.073759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:43959)
0926 19:28:12.074259 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:12.074275 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:12.074429 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:12.074790 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:12.074795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:12.075191 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:12.075346 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.075354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.076359 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.076365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.077378 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.077389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.078064 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.078069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.078246 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.078896 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:12.078914 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:12.082199 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
W20260926 19:28:12.082741 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:12.073759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:43959 (local address 127.1.39.62:41881)
0926 19:28:12.074063 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:12.074072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:12.074353 (+ 281us) server_negotiation.cc:407] Connection header received
0926 19:28:12.074563 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:12.074567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:12.074658 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:12.074759 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:12.075490 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.076224 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.077504 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.077923 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.078052 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.082418 (+ 4366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:43959: BlockingRecv error: recv got EOF from 127.1.39.1:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:12 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:12.553037 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:12.544150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:59315)
0926 19:28:12.544618 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:12.544633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:12.544801 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:12.545090 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:12.545093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:12.545451 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:12.545592 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.545600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.546629 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.546641 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.547562 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:12.547572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.548262 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.548266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:12.548398 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.549091 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:12.549110 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:12.552732 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
W20260926 19:28:12.553184 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:12.544232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:59315 (local address 127.1.39.62:41881)
0926 19:28:12.544469 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:12.544476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:12.544800 (+ 324us) server_negotiation.cc:407] Connection header received
0926 19:28:12.544907 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:12.544911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:12.544971 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:12.545055 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:12.545732 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.546486 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.547695 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:12.548128 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:12.548233 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:12.552917 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:59315: BlockingRecv error: recv got EOF from 127.1.39.2:59315 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.3@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:13.024500 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.009444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.3:56271)
0926 19:28:13.010051 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:13.010075 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:13.010242 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:13.010672 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:13.010678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:13.011129 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:13.011300 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.011311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.012480 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.012490 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.013609 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.013622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.014436 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.014443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.014645 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.021042 (+ 6397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:13.021063 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:13.024239 (+ 3176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":282,"threads_started":1}
W20260926 19:28:13.024736 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:13.009602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:56271 (local address 127.1.39.62:41881)
0926 19:28:13.009837 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.009843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.010195 (+ 352us) server_negotiation.cc:407] Connection header received
0926 19:28:13.010415 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.010423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.010511 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.010637 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.011476 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.012303 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.013767 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.014262 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.014426 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.024476 (+ 10050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.3:56271: BlockingRecv error: recv got EOF from 127.1.39.3:56271 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.1@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:13.091794 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.083591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.1:51467)
0926 19:28:13.084063 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:13.084082 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:13.084251 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:13.084616 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:13.084621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:13.085045 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:13.085202 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.085210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.086238 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.086245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.087165 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.087181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.088020 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.088026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.088198 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.088804 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:13.088821 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:13.091534 (+ 2713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":317,"threads_started":1}
W20260926 19:28:13.091917 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:13.083710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:51467 (local address 127.1.39.62:41881)
0926 19:28:13.083909 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.083916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.084175 (+ 259us) server_negotiation.cc:407] Connection header received
0926 19:28:13.084380 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.084385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.084481 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.084584 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.085361 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.086100 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.087435 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.087877 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.088008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.091701 (+ 3693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.1:51467: BlockingRecv error: recv got EOF from 127.1.39.1:51467 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 19:28:13 dist-test-slave-9wb0 krb5kdc[9395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.39.2@KRBTEST.COM for oryx/127.1.39.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 19:28:13.563103 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.554210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:41881 (local address 127.1.39.2:47799)
0926 19:28:13.554768 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:13.554785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:13.554973 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:13.555342 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:13.555347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:13.555795 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:13.555972 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.555982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.557062 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.557070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.558132 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:13.558144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.559021 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.559027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:13.559208 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.559885 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:13.559921 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:13.562730 (+ 2809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.39.62:41881: Server oryx/127.1.39.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":333,"threads_started":1}
W20260926 19:28:13.563164 10132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:13.554293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:47799 (local address 127.1.39.62:41881)
0926 19:28:13.554554 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.554563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.554883 (+ 320us) server_negotiation.cc:407] Connection header received
0926 19:28:13.555121 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.555127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.555199 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.555307 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.556127 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.556913 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.558288 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.558861 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.559010 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.562928 (+ 3918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.39.2:47799: BlockingRecv error: recv got EOF from 127.1.39.2:47799 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:28:13.846002 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9982
I20260926 19:28:13.858656 10442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.847046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.1.39.1:43321)
0926 19:28:13.847628 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.847636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.847663 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:28:13.847722 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.847729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.847982 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.848160 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.849701 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.850779 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.852031 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.852567 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.852665 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.855520 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:13.855529 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:13.855543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:13.855575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:13.858103 (+ 2528us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:28:13.858381 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35802: Request ticket server kudu/127.1.39.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
W20260926 19:28:13.858783 10442 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35802: Request ticket server kudu/127.1.39.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 19:28:13.858851 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9982: Not authorized: Client connection negotiation failed: client connection to 127.1.39.1:43321: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:28:13.859151 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9982
I20260926 19:28:13.876737 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10133
I20260926 19:28:13.889430 10443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.877996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.1.39.2:39653)
0926 19:28:13.878614 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.878621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.878653 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:28:13.878720 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.878726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.878971 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.879131 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.880358 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.881537 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.882799 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.883363 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.883473 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.886486 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:13.886501 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:13.886515 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:13.886552 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:13.888876 (+ 2324us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:28:13.889117 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47882: Request ticket server kudu/127.1.39.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":297,"threads_started":1}
W20260926 19:28:13.889544 10443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47882: Request ticket server kudu/127.1.39.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 19:28:13.889710 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10133: Not authorized: Client connection negotiation failed: client connection to 127.1.39.2:39653: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:28:13.889870 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10133
I20260926 19:28:13.907823 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10264
I20260926 19:28:13.919714 10444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:13.908816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.1.39.3:37617)
0926 19:28:13.909479 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:13.909486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:13.909512 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:28:13.909569 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:13.909575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:13.909795 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:13.909942 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:13.911276 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.912307 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.913504 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:13.914032 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:13.914130 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:13.916633 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:13.916642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:13.916655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:13.916692 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:13.919204 (+ 2512us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 19:28:13.919451 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37962: Request ticket server kudu/127.1.39.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":338,"threads_started":1}
W20260926 19:28:13.919835 10444 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37962: Request ticket server kudu/127.1.39.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 19:28:13.919917 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10264: Not authorized: Client connection negotiation failed: client connection to 127.1.39.3:37617: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.1.39.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 19:28:13.920118 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10264
I20260926 19:28:13.936767 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 9913
I20260926 19:28:13.938150 9975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:14.051990 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 9913
2026-09-26T19:28:14Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21147 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T19:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:28:14Z Disabled control of system clock
I20260926 19:28:14.130517 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46201
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 19:28:14.463721 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:14.464010 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:14.464139 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:14.471503 10449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:28:14.471612 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:14.471681 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:14.471792 10449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:28:14.471869 10449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:28:14.480955 10449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:46201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:14.482304 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:14.484299 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:14.499447 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:14.499475 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:14.500103 10449 server_base.cc:1034] running on GCE node
W20260926 19:28:14.500227 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:14.501138 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:14.502619 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:14.503846 10449 hybrid_clock.cc:648] HybridClock initialized: now 1790450894503797 us; error 65 us; skew 500 ppm
I20260926 19:28:14.504208 10449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:14.506560 10449 webserver.cc:460] Webserver started at http://127.1.39.62:43685/ using document root <none> and password file <none>
I20260926 19:28:14.507262 10449 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:14.507390 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:14.507722 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:14.511188 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "02c79780b6894b0eb916ba325a2e62c4"
format_stamp: "Formatted at 2026-09-26 19:28:14 on dist-test-slave-9wb0"
server_key: "edb8c7f913084e08786a31f9d4a26a78"
server_key_iv: "85fa94cd8cfa1435b02648cf28dc7eb7"
server_key_version: "encryptionkey@0"
I20260926 19:28:14.512115 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "02c79780b6894b0eb916ba325a2e62c4"
format_stamp: "Formatted at 2026-09-26 19:28:14 on dist-test-slave-9wb0"
server_key: "edb8c7f913084e08786a31f9d4a26a78"
server_key_iv: "85fa94cd8cfa1435b02648cf28dc7eb7"
server_key_version: "encryptionkey@0"
I20260926 19:28:14.517976 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260926 19:28:14.521739 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:14.523268 10449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:28:14.523471 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "02c79780b6894b0eb916ba325a2e62c4"
format_stamp: "Formatted at 2026-09-26 19:28:14 on dist-test-slave-9wb0"
server_key: "edb8c7f913084e08786a31f9d4a26a78"
server_key_iv: "85fa94cd8cfa1435b02648cf28dc7eb7"
server_key_version: "encryptionkey@0"
I20260926 19:28:14.523715 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:14.542860 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:14.543742 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:14.544034 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:14.563982 10449 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:46201
I20260926 19:28:14.564015 10514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:46201 every 8 connection(s)
I20260926 19:28:14.565490 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:28:14.569370 10515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:28:14.572999 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10449
I20260926 19:28:14.573251 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:28:14.574123 1180 external_mini_cluster.cc:1428] Setting key c792edd33922642252401bd3fe884052
I20260926 19:28:14.581169 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4: Bootstrap starting.
I20260926 19:28:14.584806 10515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:14.586040 10515 log.cc:824] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:14.589159 10515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4: No bootstrap required, opened a new log
I20260926 19:28:14.589716 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:14.576226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.1.39.62:46201)
0926 19:28:14.577415 (+ 1189us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:14.577427 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:14.577473 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:28:14.579871 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:14.579896 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:14.580397 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:14.580749 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:14.581447 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:14.582889 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:14.584759 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:14.585634 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:14.585779 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:14.587166 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:14.587179 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:14.587204 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:14.587888 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:14.587960 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:14.587985 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:14.588089 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:14.588335 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 19:28:14.588676 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":422,"threads_started":1}
I20260926 19:28:14.596462 10515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } }
I20260926 19:28:14.596817 10515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:14.596920 10515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c79780b6894b0eb916ba325a2e62c4, State: Initialized, Role: FOLLOWER
I20260926 19:28:14.597568 10515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } }
I20260926 19:28:14.597787 10515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:28:14.597931 10515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:28:14.598109 10515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:14.600273 10515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } }
I20260926 19:28:14.600754 10515 leader_election.cc:302] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02c79780b6894b0eb916ba325a2e62c4; no voters:
I20260926 19:28:14.601365 10515 leader_election.cc:288] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:28:14.601552 10520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:28:14.602250 10520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [term 1 LEADER]: Becoming Leader. State: Replica: 02c79780b6894b0eb916ba325a2e62c4, State: Running, Role: LEADER
I20260926 19:28:14.602797 10520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } }
I20260926 19:28:14.603434 10515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:28:14.605576 10521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02c79780b6894b0eb916ba325a2e62c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } } }
I20260926 19:28:14.605926 10521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:28:14.606112 10522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c79780b6894b0eb916ba325a2e62c4. Latest consensus state: current_term: 1 leader_uuid: "02c79780b6894b0eb916ba325a2e62c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02c79780b6894b0eb916ba325a2e62c4" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46201 } } }
I20260926 19:28:14.606427 10522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4 [sys.catalog]: This master's current role is: LEADER
I20260926 19:28:14.607959 10527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:28:14.613461 10527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:28:14.622113 10527 catalog_manager.cc:1347] Generated new cluster ID: bbe72f27aea443a690e95a514e5df360
I20260926 19:28:14.622221 10527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:28:14.697829 10527 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:28:14.699294 10527 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:28:14.707763 10527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02c79780b6894b0eb916ba325a2e62c4: Generated new TSK 0
I20260926 19:28:14.708848 10527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:28:14.724265 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--builtin_ntp_servers=127.1.39.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 19:28:15.060631 10539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:15.060922 10539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:15.061004 10539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:15.068507 10539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:15.068620 10539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:15.068723 10539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:28:15.077877 10539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:15.079108 10539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:15.080794 10539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:15.094707 10544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:15.094894 10545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:15.095216 10547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:15.095907 10539 server_base.cc:1034] running on GCE node
I20260926 19:28:15.096596 10539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:15.097572 10539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:15.098788 10539 hybrid_clock.cc:648] HybridClock initialized: now 1790450895098736 us; error 45 us; skew 500 ppm
I20260926 19:28:15.099104 10539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:15.101496 10539 webserver.cc:460] Webserver started at http://127.1.39.1:34587/ using document root <none> and password file <none>
I20260926 19:28:15.102185 10539 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:15.102308 10539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:15.102615 10539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:15.105644 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "9302e2faee3d4906878618c2b9618443"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "579e78d362603c30a957af6f0e1aa10a"
server_key_iv: "6c3d4334fb7398a292a5b9a27dc61210"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.106401 10539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "9302e2faee3d4906878618c2b9618443"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "579e78d362603c30a957af6f0e1aa10a"
server_key_iv: "6c3d4334fb7398a292a5b9a27dc61210"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.111588 10539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 19:28:15.115078 10553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.116405 10539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:28:15.116578 10539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "9302e2faee3d4906878618c2b9618443"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "579e78d362603c30a957af6f0e1aa10a"
server_key_iv: "6c3d4334fb7398a292a5b9a27dc61210"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.116786 10539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:15.144332 10539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:15.145226 10539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:15.145525 10539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:15.146456 10539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:15.147845 10539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:15.147962 10539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.148095 10539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:15.148185 10539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.180400 10539 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:37853
I20260926 19:28:15.180485 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:37853 every 8 connection(s)
I20260926 19:28:15.181669 10539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:28:15.189471 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10539
I20260926 19:28:15.189709 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:28:15.190361 1180 external_mini_cluster.cc:1428] Setting key 7db452f9484a161a837d854524308b20
I20260926 19:28:15.193195 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--builtin_ntp_servers=127.1.39.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 19:28:15.196437 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:15.184155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:60339 (local address 127.1.39.62:46201)
0926 19:28:15.184633 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:15.184637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:15.186409 (+ 1772us) server_negotiation.cc:407] Connection header received
0926 19:28:15.188760 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:15.188768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:15.188870 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:15.189006 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:15.190425 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.191397 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.193273 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.193848 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.193997 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:15.195488 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:15.195494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:15.195497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:15.195517 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:15.195563 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:15.195576 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:15.195680 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:15.195928 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 19:28:15.196037 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260926 19:28:15.197690 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:15.184615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46201 (local address 127.1.39.1:60339)
0926 19:28:15.186195 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:15.186244 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:15.188277 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:15.189274 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:15.189284 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:15.189462 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:15.190201 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:15.190221 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.191567 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.191573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:15.193105 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:15.193118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.194092 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.194100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:15.194347 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:15.195110 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:15.195113 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:15.195161 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:15.195248 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:15.195691 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:15.195704 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:15.195718 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 19:28:15.195948 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 19:28:15.196374 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":455,"threads_started":1}
I20260926 19:28:15.199524 10666 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46201
I20260926 19:28:15.199867 10666 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:15.200837 10666 heartbeater.cc:511] Master 127.1.39.62:46201 requested a full tablet report, sending...
I20260926 19:28:15.203291 10480 ts_manager.cc:194] Registered new tserver with Master: 9302e2faee3d4906878618c2b9618443 (127.1.39.1:37853)
I20260926 19:28:15.205093 10480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.1:60339
W20260926 19:28:15.528663 10670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:15.528947 10670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:15.529088 10670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:15.536566 10670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:15.536676 10670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:15.536852 10670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:28:15.546034 10670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:15.547345 10670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:15.549230 10670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:15.562467 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:15.562389 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:15.562440 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:15.563825 10670 server_base.cc:1034] running on GCE node
I20260926 19:28:15.564500 10670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:15.565706 10670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:15.566931 10670 hybrid_clock.cc:648] HybridClock initialized: now 1790450895566864 us; error 54 us; skew 500 ppm
I20260926 19:28:15.567286 10670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:15.570029 10670 webserver.cc:460] Webserver started at http://127.1.39.2:39261/ using document root <none> and password file <none>
I20260926 19:28:15.570874 10670 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:15.570983 10670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:15.571298 10670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:15.574936 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "f33b1362341247f388876ae55988f99d"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "10c61fb7bef949059b88902e50e37980"
server_key_iv: "0bc457ba596a1cc36f408fa20d676353"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.575922 10670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "f33b1362341247f388876ae55988f99d"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "10c61fb7bef949059b88902e50e37980"
server_key_iv: "0bc457ba596a1cc36f408fa20d676353"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.582161 10670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 19:28:15.586052 10684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.587342 10670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:28:15.587504 10670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "f33b1362341247f388876ae55988f99d"
format_stamp: "Formatted at 2026-09-26 19:28:15 on dist-test-slave-9wb0"
server_key: "10c61fb7bef949059b88902e50e37980"
server_key_iv: "0bc457ba596a1cc36f408fa20d676353"
server_key_version: "encryptionkey@0"
I20260926 19:28:15.587706 10670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:15.623661 10670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:15.624532 10670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:15.624821 10670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:15.625871 10670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:15.627344 10670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:15.627480 10670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.627569 10670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:15.627625 10670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:15.660579 10670 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:44475
I20260926 19:28:15.660633 10796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:44475 every 8 connection(s)
I20260926 19:28:15.661933 10670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:28:15.671820 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10670
I20260926 19:28:15.672010 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:28:15.672627 1180 external_mini_cluster.cc:1428] Setting key 3aec359d94d3632fb1a2ba047ac953aa
I20260926 19:28:15.674221 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:15.664476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:43221 (local address 127.1.39.62:46201)
0926 19:28:15.664647 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:15.664651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:15.666140 (+ 1489us) server_negotiation.cc:407] Connection header received
0926 19:28:15.667814 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:15.667819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:15.667886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:15.667999 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:15.668972 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.669767 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.671250 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.671675 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.671775 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:15.673418 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:15.673424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:15.673428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:15.673449 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:15.673486 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:15.673502 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:15.673598 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:15.673917 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:28:15.673991 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:28:15.675450 10799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:15.664908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46201 (local address 127.1.39.2:43221)
0926 19:28:15.665984 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:15.666025 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:15.667546 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:15.668180 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:15.668187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:15.668269 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:15.668820 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:15.668836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.669896 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.669900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:15.671112 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:15.671123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:15.671903 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:15.671911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:15.672232 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:15.673025 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:15.673028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:15.673065 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:15.673185 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:15.673660 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:15.673672 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:15.673683 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:28:15.673918 (+ 235us) client_negotiation.cc:240] Negotiation successful
0926 19:28:15.674307 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":381,"threads_started":1}
I20260926 19:28:15.675503 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--builtin_ntp_servers=127.1.39.20:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 19:28:15.676894 10797 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46201
I20260926 19:28:15.677214 10797 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:15.678130 10797 heartbeater.cc:511] Master 127.1.39.62:46201 requested a full tablet report, sending...
I20260926 19:28:15.679597 10480 ts_manager.cc:194] Registered new tserver with Master: f33b1362341247f388876ae55988f99d (127.1.39.2:44475)
I20260926 19:28:15.680385 10480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.2:43221
W20260926 19:28:16.006491 10800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:16.006820 10800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:16.006968 10800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:16.014357 10800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:16.014492 10800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:16.014662 10800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:28:16.023943 10800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:16.025213 10800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:16.026964 10800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:16.039813 10808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:16.039892 10805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:16.039865 10806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:16.040702 10800 server_base.cc:1034] running on GCE node
I20260926 19:28:16.041277 10800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:16.042299 10800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:16.043540 10800 hybrid_clock.cc:648] HybridClock initialized: now 1790450896043474 us; error 67 us; skew 500 ppm
I20260926 19:28:16.043859 10800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:16.046164 10800 webserver.cc:460] Webserver started at http://127.1.39.3:37567/ using document root <none> and password file <none>
I20260926 19:28:16.046855 10800 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:16.046986 10800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:16.047304 10800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:16.050272 10800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "707bf134b6e84ec481b41f85dc5e978f"
format_stamp: "Formatted at 2026-09-26 19:28:16 on dist-test-slave-9wb0"
server_key: "3d3f400494a1a71408ab520d6e4dbe29"
server_key_iv: "cf19e04f26f2934a952647e55575c27f"
server_key_version: "encryptionkey@0"
I20260926 19:28:16.051096 10800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "707bf134b6e84ec481b41f85dc5e978f"
format_stamp: "Formatted at 2026-09-26 19:28:16 on dist-test-slave-9wb0"
server_key: "3d3f400494a1a71408ab520d6e4dbe29"
server_key_iv: "cf19e04f26f2934a952647e55575c27f"
server_key_version: "encryptionkey@0"
I20260926 19:28:16.056202 10800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:28:16.059821 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:16.061069 10800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:28:16.061246 10800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "707bf134b6e84ec481b41f85dc5e978f"
format_stamp: "Formatted at 2026-09-26 19:28:16 on dist-test-slave-9wb0"
server_key: "3d3f400494a1a71408ab520d6e4dbe29"
server_key_iv: "cf19e04f26f2934a952647e55575c27f"
server_key_version: "encryptionkey@0"
I20260926 19:28:16.061534 10800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:16.108897 10800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:16.109901 10800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:16.110199 10800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:16.111227 10800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:16.112731 10800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:16.112850 10800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:16.112983 10800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:16.113067 10800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:16.145706 10800 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:46669
I20260926 19:28:16.145730 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:46669 every 8 connection(s)
I20260926 19:28:16.146992 10800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:28:16.151332 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10800
I20260926 19:28:16.151494 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:28:16.151923 1180 external_mini_cluster.cc:1428] Setting key 17156a2ebe8b8d3e2281782744679403
I20260926 19:28:16.159252 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:16.149544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:45349 (local address 127.1.39.62:46201)
0926 19:28:16.149723 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:16.149728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:16.151121 (+ 1393us) server_negotiation.cc:407] Connection header received
0926 19:28:16.153009 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:16.153016 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:16.153096 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:16.153215 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:16.154506 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.155406 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.156858 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.157239 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.157385 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:16.158622 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:16.158642 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:16.158645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:16.158662 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:16.158689 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:16.158700 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:16.158768 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:16.158954 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 19:28:16.159028 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:28:16.160184 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:16.150088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46201 (local address 127.1.39.3:45349)
0926 19:28:16.150957 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:16.150998 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:16.152710 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:16.153454 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:16.153462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:16.153570 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:16.154353 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:16.154376 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.155533 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.155536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:16.156742 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:16.156753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.157452 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.157457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:16.157680 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:16.158353 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:16.158355 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:16.158392 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:16.158461 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:16.158780 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:16.158787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:16.158792 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:28:16.158934 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 19:28:16.159205 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260926 19:28:16.161659 10927 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46201
I20260926 19:28:16.161902 10927 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:16.162540 10927 heartbeater.cc:511] Master 127.1.39.62:46201 requested a full tablet report, sending...
I20260926 19:28:16.163888 10480 ts_manager.cc:194] Registered new tserver with Master: 707bf134b6e84ec481b41f85dc5e978f (127.1.39.3:46669)
I20260926 19:28:16.164726 10480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.3:45349
I20260926 19:28:16.167091 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 19:28:16.173229 10668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:16.171469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49238 (local address 127.1.39.62:46201)
0926 19:28:16.171716 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:16.171721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:16.172205 (+ 484us) server_negotiation.cc:407] Connection header received
0926 19:28:16.172343 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:16.172347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:16.172449 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:16.172549 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:16.172948 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49238: BlockingRecv error: recv got EOF from 127.0.0.1:49238 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 19:28:16.174145 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10539
I20260926 19:28:16.181995 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:16.175062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.1.39.1:37853)
0926 19:28:16.175560 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:16.175576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:16.175600 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:28:16.175658 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:16.175664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:16.175863 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:16.175986 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:16.176407 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.177432 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.178712 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.179235 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.179333 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:16.180425 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:16.180435 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:16.180460 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:16.181036 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:16.181091 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:16.181112 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:16.181236 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:16.181445 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 19:28:16.181581 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":270,"threads_started":1}
I20260926 19:28:16.183362 10661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:16.264777 10666 heartbeater.cc:503] Master 127.1.39.62:46201 was elected leader, sending a full tablet report...
I20260926 19:28:16.266408 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10539
W20260926 19:28:16.284597 10464 connection.cc:535] server connection from 127.1.39.1:60339 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:28:16.284780 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10670
I20260926 19:28:16.292822 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:16.285800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.1.39.2:44475)
0926 19:28:16.286334 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:16.286341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:16.286368 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:28:16.286426 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:16.286432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:16.286660 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:16.286783 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:16.287144 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.288154 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.289536 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.290087 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.290203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:16.291274 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:16.291282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:16.291296 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:16.291956 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:16.292001 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:16.292022 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:16.292111 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:16.292336 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 19:28:16.292497 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":287,"threads_started":1}
I20260926 19:28:16.294178 10792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:16.376255 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10670
I20260926 19:28:16.392748 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10800
I20260926 19:28:16.400911 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:16.393951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.1.39.3:46669)
0926 19:28:16.394641 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:16.394663 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:16.394697 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:28:16.394758 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:16.394765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:16.394990 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:16.395127 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:16.395559 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.396624 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.397947 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:16.398510 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:16.398609 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:16.399532 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:16.399541 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:16.399554 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:16.400141 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:16.400189 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:16.400208 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:16.400295 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:16.400388 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 19:28:16.400495 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":382,"threads_started":1}
I20260926 19:28:16.402263 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:16.488499 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10800
I20260926 19:28:16.505538 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10449
I20260926 19:28:16.506682 10510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:16.607336 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10449
2026-09-26T19:28:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2550 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T19:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:28:16Z Disabled control of system clock
I20260926 19:28:16.682267 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42491
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 19:28:17.023936 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:17.024248 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:17.024340 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:17.031726 10946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:28:17.031858 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:17.031903 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:17.031972 10946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:28:17.032019 10946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:28:17.041822 10946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42491
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:17.043174 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:17.045006 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:17.057394 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:17.057521 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:17.057567 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:17.058837 10946 server_base.cc:1034] running on GCE node
I20260926 19:28:17.059731 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:17.061144 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:17.062399 10946 hybrid_clock.cc:648] HybridClock initialized: now 1790450897062336 us; error 51 us; skew 500 ppm
I20260926 19:28:17.062842 10946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:17.065200 10946 webserver.cc:460] Webserver started at http://127.1.39.62:40321/ using document root <none> and password file <none>
I20260926 19:28:17.065984 10946 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:17.066121 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:17.066464 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:17.069550 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "4df00a498ade4bf882f51e0f462afa3d"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "919a6998bd58e5a71ae088453528e01e"
server_key_iv: "adf31050bdc0c2b6614b8475ac317055"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.070387 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "4df00a498ade4bf882f51e0f462afa3d"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "919a6998bd58e5a71ae088453528e01e"
server_key_iv: "adf31050bdc0c2b6614b8475ac317055"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.076346 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 19:28:17.080025 10960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:17.081480 10946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:28:17.081686 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "4df00a498ade4bf882f51e0f462afa3d"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "919a6998bd58e5a71ae088453528e01e"
server_key_iv: "adf31050bdc0c2b6614b8475ac317055"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.081902 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:17.087937 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:17.088699 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:17.089002 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:17.107841 10946 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42491
I20260926 19:28:17.107867 11011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42491 every 8 connection(s)
I20260926 19:28:17.109186 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:28:17.112896 11012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:28:17.114446 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 10946
I20260926 19:28:17.114684 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:28:17.115309 1180 external_mini_cluster.cc:1428] Setting key bbb043b29772cf8d30caa26f1f02ca34
I20260926 19:28:17.123576 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:17.117330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.1.39.62:42491)
0926 19:28:17.118403 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:17.118414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:17.118462 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:28:17.120061 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:17.120084 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:17.120497 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:17.120823 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:17.121953 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:17.121959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:17.121970 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:17.121987 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:17.122044 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:17.122063 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:17.122253 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:17.122352 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 19:28:17.122682 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":385,"threads_started":1}
I20260926 19:28:17.124708 11012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d: Bootstrap starting.
I20260926 19:28:17.127712 11012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:17.128679 11012 log.cc:824] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:17.131474 11012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d: No bootstrap required, opened a new log
I20260926 19:28:17.138681 11012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } }
I20260926 19:28:17.139077 11012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:17.139194 11012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df00a498ade4bf882f51e0f462afa3d, State: Initialized, Role: FOLLOWER
I20260926 19:28:17.139757 11012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } }
I20260926 19:28:17.139988 11012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:28:17.140129 11012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:28:17.140296 11012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:17.142491 11012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } }
I20260926 19:28:17.142992 11012 leader_election.cc:302] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df00a498ade4bf882f51e0f462afa3d; no voters:
I20260926 19:28:17.143667 11012 leader_election.cc:288] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:28:17.143808 11017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:28:17.144562 11017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [term 1 LEADER]: Becoming Leader. State: Replica: 4df00a498ade4bf882f51e0f462afa3d, State: Running, Role: LEADER
I20260926 19:28:17.145147 11017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } }
I20260926 19:28:17.145802 11012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:28:17.147868 11018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df00a498ade4bf882f51e0f462afa3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } } }
I20260926 19:28:17.148195 11018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [sys.catalog]: This master's current role is: LEADER
I20260926 19:28:17.149124 11024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:28:17.148984 11019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df00a498ade4bf882f51e0f462afa3d. Latest consensus state: current_term: 1 leader_uuid: "4df00a498ade4bf882f51e0f462afa3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df00a498ade4bf882f51e0f462afa3d" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42491 } } }
I20260926 19:28:17.149641 11019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d [sys.catalog]: This master's current role is: LEADER
I20260926 19:28:17.155972 11024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:28:17.167591 11024 catalog_manager.cc:1347] Generated new cluster ID: 69f09f1139d54926b3ec4ba840a344ac
I20260926 19:28:17.167862 11024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:28:17.212879 11024 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:28:17.213505 11024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:28:17.225971 11024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df00a498ade4bf882f51e0f462afa3d: Generated new TSK 0
I20260926 19:28:17.227137 11024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:28:17.246052 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--builtin_ntp_servers=127.1.39.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 19:28:17.604349 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:17.604792 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:17.604950 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:17.612448 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:17.612586 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:17.612769 11036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:28:17.623682 11036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:17.625077 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:17.627041 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:17.642248 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:17.642333 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:17.642354 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:17.646989 11036 server_base.cc:1034] running on GCE node
I20260926 19:28:17.647730 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:17.649008 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:17.650244 11036 hybrid_clock.cc:648] HybridClock initialized: now 1790450897650172 us; error 67 us; skew 500 ppm
I20260926 19:28:17.650620 11036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:17.653138 11036 webserver.cc:460] Webserver started at http://127.1.39.1:39987/ using document root <none> and password file <none>
I20260926 19:28:17.653885 11036 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:17.653988 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:17.654352 11036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:17.659075 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "093719efaeef46308b2ba3fe8e061f65"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "3e4085345dba9aa64549610d26bd19e8"
server_key_iv: "426edfcc03bf4e8000a1f55d5b03fc9f"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.659898 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "093719efaeef46308b2ba3fe8e061f65"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "3e4085345dba9aa64549610d26bd19e8"
server_key_iv: "426edfcc03bf4e8000a1f55d5b03fc9f"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.666708 11036 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 19:28:17.670956 11050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:17.672550 11036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:28:17.672772 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "093719efaeef46308b2ba3fe8e061f65"
format_stamp: "Formatted at 2026-09-26 19:28:17 on dist-test-slave-9wb0"
server_key: "3e4085345dba9aa64549610d26bd19e8"
server_key_iv: "426edfcc03bf4e8000a1f55d5b03fc9f"
server_key_version: "encryptionkey@0"
I20260926 19:28:17.673050 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:17.680202 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:17.681075 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:17.681423 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:17.682399 11036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:17.683871 11036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:17.684007 11036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:17.684144 11036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:17.684232 11036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:17.720541 11036 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:34363
I20260926 19:28:17.720631 11162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:34363 every 8 connection(s)
I20260926 19:28:17.721971 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:28:17.723966 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11036
I20260926 19:28:17.724188 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:28:17.724819 1180 external_mini_cluster.cc:1428] Setting key 146aaf1e7790b08c6f634b270c9733c2
I20260926 19:28:17.728034 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--builtin_ntp_servers=127.1.39.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 19:28:17.731516 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:17.724983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:54937 (local address 127.1.39.62:42491)
0926 19:28:17.725462 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:17.725468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:17.727072 (+ 1604us) server_negotiation.cc:407] Connection header received
0926 19:28:17.728923 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:17.728934 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:17.729013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:17.729163 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:17.730564 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:17.730573 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:17.730582 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:17.730609 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:17.730671 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:17.730693 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:17.730850 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:17.731001 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 19:28:17.731074 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":219,"threads_started":1}
I20260926 19:28:17.732179 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:17.725663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42491 (local address 127.1.39.1:54937)
0926 19:28:17.726844 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:17.726903 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:17.728528 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:17.729414 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:17.729422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:17.729519 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:17.730253 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:17.730255 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:17.730292 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:17.730376 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:17.730802 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:17.730809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:17.730814 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:28:17.731019 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 19:28:17.731230 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":362,"threads_started":1}
I20260926 19:28:17.733764 11163 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42491
I20260926 19:28:17.734059 11163 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:17.734653 11163 heartbeater.cc:511] Master 127.1.39.62:42491 requested a full tablet report, sending...
I20260926 19:28:17.736964 10977 ts_manager.cc:194] Registered new tserver with Master: 093719efaeef46308b2ba3fe8e061f65 (127.1.39.1:34363)
W20260926 19:28:18.072741 11167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:18.073019 11167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:18.073117 11167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:18.080375 11167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:18.080477 11167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:18.080586 11167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:28:18.089720 11167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:18.090947 11167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:18.092628 11167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:18.105201 11172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:18.105252 11173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:18.105575 11175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:18.106247 11167 server_base.cc:1034] running on GCE node
I20260926 19:28:18.107004 11167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:18.108110 11167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:18.109380 11167 hybrid_clock.cc:648] HybridClock initialized: now 1790450898109265 us; error 126 us; skew 500 ppm
I20260926 19:28:18.109709 11167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:18.119033 11167 webserver.cc:460] Webserver started at http://127.1.39.2:45721/ using document root <none> and password file <none>
I20260926 19:28:18.119951 11167 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:18.120087 11167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:18.120412 11167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:18.124104 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "1c9c5fab9743413ea72c8800bc8dc30b"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "b72d0404196cd924e89527477bd271ba"
server_key_iv: "49d77b6670bab8ade6be781e0f922ab2"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.124847 11167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "1c9c5fab9743413ea72c8800bc8dc30b"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "b72d0404196cd924e89527477bd271ba"
server_key_iv: "49d77b6670bab8ade6be781e0f922ab2"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.129791 11167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:28:18.134766 11181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.136341 11167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:28:18.136497 11167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "1c9c5fab9743413ea72c8800bc8dc30b"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "b72d0404196cd924e89527477bd271ba"
server_key_iv: "49d77b6670bab8ade6be781e0f922ab2"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.136699 11167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:18.143115 11167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:18.143857 11167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:18.144078 11167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:18.144918 11167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:18.146534 11167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:18.146628 11167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.146711 11167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:18.146759 11167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.181165 11167 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:35601
I20260926 19:28:18.181223 11293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:35601 every 8 connection(s)
I20260926 19:28:18.182617 11167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:28:18.187003 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11167
I20260926 19:28:18.187206 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:28:18.187796 1180 external_mini_cluster.cc:1428] Setting key 9d072e2e3346f30ec2bf0d6d51f85b90
I20260926 19:28:18.191206 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--builtin_ntp_servers=127.1.39.20:41099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 19:28:18.191864 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.185429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:45167 (local address 127.1.39.62:42491)
0926 19:28:18.185623 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.185628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.187270 (+ 1642us) server_negotiation.cc:407] Connection header received
0926 19:28:18.189390 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.189396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.189460 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.189567 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.191135 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:18.191140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:18.191144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:18.191161 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:18.191192 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:18.191205 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:18.191368 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:18.191585 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 19:28:18.191648 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:28:18.193254 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.185932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42491 (local address 127.1.39.2:45167)
0926 19:28:18.187074 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:18.187135 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:18.189099 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:18.189849 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:18.189857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:18.189983 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:18.190732 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:18.190736 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:18.190799 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:18.190922 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:18.191320 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:18.191331 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:18.191342 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:28:18.191644 (+ 302us) client_negotiation.cc:240] Negotiation successful
0926 19:28:18.191909 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":315,"threads_started":1}
I20260926 19:28:18.194690 11294 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42491
I20260926 19:28:18.195024 11294 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:18.195712 11294 heartbeater.cc:511] Master 127.1.39.62:42491 requested a full tablet report, sending...
I20260926 19:28:18.197343 10977 ts_manager.cc:194] Registered new tserver with Master: 1c9c5fab9743413ea72c8800bc8dc30b (127.1.39.2:35601)
W20260926 19:28:18.532033 11297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:18.532312 11297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:18.532464 11297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:18.539727 11297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:18.539832 11297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:18.539999 11297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:28:18.549779 11297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:18.551344 11297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:18.553220 11297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:18.566424 11303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:18.566536 11302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:18.566493 11305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:18.567687 11297 server_base.cc:1034] running on GCE node
I20260926 19:28:18.568284 11297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:18.569267 11297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:18.570574 11297 hybrid_clock.cc:648] HybridClock initialized: now 1790450898570520 us; error 35 us; skew 500 ppm
I20260926 19:28:18.571004 11297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:28:18.573745 11297 webserver.cc:460] Webserver started at http://127.1.39.3:46441/ using document root <none> and password file <none>
I20260926 19:28:18.574527 11297 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:18.574661 11297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:18.575073 11297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:18.578329 11297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "4a0adce98fd040819532afdcf21c3afd"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "10e6b3f3f18dd6900c7ecfff26719cd9"
server_key_iv: "c0baf7ecd66f85fd0a332eb910368072"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.579244 11297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "4a0adce98fd040819532afdcf21c3afd"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "10e6b3f3f18dd6900c7ecfff26719cd9"
server_key_iv: "c0baf7ecd66f85fd0a332eb910368072"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.584895 11297 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:28:18.588632 11311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.590111 11297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:28:18.590293 11297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "4a0adce98fd040819532afdcf21c3afd"
format_stamp: "Formatted at 2026-09-26 19:28:18 on dist-test-slave-9wb0"
server_key: "10e6b3f3f18dd6900c7ecfff26719cd9"
server_key_iv: "c0baf7ecd66f85fd0a332eb910368072"
server_key_version: "encryptionkey@0"
I20260926 19:28:18.590490 11297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:18.596170 11297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:18.596930 11297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:18.597199 11297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:18.598165 11297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:18.599498 11297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:18.599583 11297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.599728 11297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:18.599799 11297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:18.635668 11297 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:43235
I20260926 19:28:18.635784 11423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:43235 every 8 connection(s)
I20260926 19:28:18.636965 11297 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:28:18.645638 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.639567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:47743 (local address 127.1.39.62:42491)
0926 19:28:18.639776 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.639782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.641495 (+ 1713us) server_negotiation.cc:407] Connection header received
0926 19:28:18.643475 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.643480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.643541 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.643630 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.644933 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:18.644937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:18.644941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:18.644957 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:18.644986 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:18.644997 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:18.645088 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:18.645360 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 19:28:18.645410 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:28:18.646672 11426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.640113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42491 (local address 127.1.39.3:47743)
0926 19:28:18.641245 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:18.641334 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:18.643178 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:18.643876 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:18.643886 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:18.643997 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:18.644647 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:28:18.644650 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:18.644694 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:28:18.644777 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:18.645111 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:18.645119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:18.645125 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 19:28:18.645436 (+ 311us) client_negotiation.cc:240] Negotiation successful
0926 19:28:18.645631 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":349,"threads_started":1}
I20260926 19:28:18.647351 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11297
I20260926 19:28:18.647498 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:28:18.647915 1180 external_mini_cluster.cc:1428] Setting key 3acc99d9dba7fcba2654e5d50c5bb6f3
I20260926 19:28:18.648139 11424 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42491
I20260926 19:28:18.648452 11424 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:18.649111 11424 heartbeater.cc:511] Master 127.1.39.62:42491 requested a full tablet report, sending...
I20260926 19:28:18.650604 10976 ts_manager.cc:194] Registered new tserver with Master: 4a0adce98fd040819532afdcf21c3afd (127.1.39.3:43235)
I20260926 19:28:18.662933 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 19:28:18.668787 11165 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:28:18.667436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.1.39.62:42491)
0926 19:28:18.667607 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.667612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.668043 (+ 431us) server_negotiation.cc:407] Connection header received
0926 19:28:18.668165 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.668169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.668220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.668306 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.668590 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55940: BlockingRecv error: recv got EOF from 127.0.0.1:55940 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:28:18.669215 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11036
I20260926 19:28:18.672708 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.670230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.1.39.1:34363)
0926 19:28:18.670760 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.670766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.670791 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:28:18.670849 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.670854 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.671044 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.671165 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.672019 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:18.672025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:18.672036 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:18.672054 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:18.672094 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:18.672108 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:18.672209 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:18.672282 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 19:28:18.672396 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":287,"threads_started":1}
I20260926 19:28:18.673991 11158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:18.754206 11163 heartbeater.cc:503] Master 127.1.39.62:42491 was elected leader, sending a full tablet report...
I20260926 19:28:18.755636 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11036
W20260926 19:28:18.774017 10961 connection.cc:535] server connection from 127.1.39.1:54937 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:28:18.774288 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11167
I20260926 19:28:18.777894 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.775207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58552 (local address 127.1.39.2:35601)
0926 19:28:18.775714 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.775721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.775747 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:28:18.775804 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.775811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.776032 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.776198 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.777191 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:18.777197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:18.777208 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:18.777224 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:18.777261 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:18.777274 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:18.777450 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:18.777525 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 19:28:18.777623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":270,"threads_started":1}
I20260926 19:28:18.779214 11289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:18.862316 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11167
I20260926 19:28:18.879292 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11297
I20260926 19:28:18.883385 11437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:18.880387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.1.39.3:43235)
0926 19:28:18.881103 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:18.881110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:18.881140 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:28:18.881199 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:18.881205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:18.881529 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:18.881689 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:18.882559 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:18.882565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:18.882586 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:28:18.882600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:18.882637 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:28:18.882649 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:18.882777 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:18.882958 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:28:18.883069 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":406,"threads_started":1}
I20260926 19:28:18.884697 11419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:18.975389 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11297
I20260926 19:28:19.001031 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 10946
I20260926 19:28:19.002388 11007 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:19.102697 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 10946
2026-09-26T19:28:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2494 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:28:19 dist-test-slave-9wb0 krb5kdc[11442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:28:19 dist-test-slave-9wb0 krb5kdc[11442](info): set up 2 sockets
Sep 26 19:28:19 dist-test-slave-9wb0 krb5kdc[11442](info): commencing operation
krb5kdc: starting...
W20260926 19:28:20.214323 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:28:20 dist-test-slave-9wb0 krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:28:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:28:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:28:20.458069 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37569
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:28:20.782284 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:20.782557 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:20.782714 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:20.789650 11455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:28:20.789758 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:20.789873 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:20.789959 11455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:28:20.790035 11455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:28:20.799796 11455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:20.801079 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:20.802749 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:20.817144 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:20.817054 11463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:20.817102 11461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:20.818215 11455 server_base.cc:1034] running on GCE node
I20260926 19:28:20.819057 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:20.820556 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:20.821794 11455 hybrid_clock.cc:648] HybridClock initialized: now 1790450900821739 us; error 34 us; skew 500 ppm
I20260926 19:28:20.822189 11455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:28:20 dist-test-slave-9wb0 krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:28:20.825351 11455 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:28:20.827097 11455 webserver.cc:460] Webserver started at http://127.1.39.62:39253/ using document root <none> and password file <none>
I20260926 19:28:20.827791 11455 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:20.827956 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:20.828274 11455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:20.831346 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "01e1a33390e04e4c9447bd4ea9092099"
format_stamp: "Formatted at 2026-09-26 19:28:20 on dist-test-slave-9wb0"
server_key: "aa85e35572cf210a7b151ae902790ffb"
server_key_iv: "4bc06479eae5ce582b1418aafa2ecf02"
server_key_version: "encryptionkey@0"
I20260926 19:28:20.832165 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "01e1a33390e04e4c9447bd4ea9092099"
format_stamp: "Formatted at 2026-09-26 19:28:20 on dist-test-slave-9wb0"
server_key: "aa85e35572cf210a7b151ae902790ffb"
server_key_iv: "4bc06479eae5ce582b1418aafa2ecf02"
server_key_version: "encryptionkey@0"
I20260926 19:28:20.837522 11455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:28:20.840984 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:20.842372 11455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:28:20.842567 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "01e1a33390e04e4c9447bd4ea9092099"
format_stamp: "Formatted at 2026-09-26 19:28:20 on dist-test-slave-9wb0"
server_key: "aa85e35572cf210a7b151ae902790ffb"
server_key_iv: "4bc06479eae5ce582b1418aafa2ecf02"
server_key_version: "encryptionkey@0"
I20260926 19:28:20.842772 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:20.864624 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:20.868645 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:20.868964 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:20.888144 11455 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:37569
I20260926 19:28:20.888180 11521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:37569 every 8 connection(s)
I20260926 19:28:20.889572 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:28:20.891494 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11455
I20260926 19:28:20.891721 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:28:20.892491 1180 external_mini_cluster.cc:1428] Setting key 80afc97f58e50b20513f30c3285325d1
I20260926 19:28:20.894577 11522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:28:20 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:28:20.908561 11522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: Bootstrap starting.
I20260926 19:28:20.912051 11522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:20.913051 11522 log.cc:824] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:20.915907 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:20.894605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.1.39.62:37569)
0926 19:28:20.895428 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:20.895440 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:20.895494 (+ 54us) server_negotiation.cc:407] Connection header received
0926 19:28:20.897087 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:20.897109 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:20.897586 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:20.897937 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:20.899790 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:20.901275 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:20.903061 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:20.903758 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:20.903900 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:20.908583 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:20.908593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:20.908610 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:20.908655 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:20.911759 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:20.912548 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:20.912555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:20.912562 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:20.912740 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:20.913131 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:20.913137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:20.913140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:20.913796 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:20.914131 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:20.914550 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 19:28:20.914979 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":371,"threads_started":1}
I20260926 19:28:20.915993 11522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: No bootstrap required, opened a new log
I20260926 19:28:20.923753 11522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } }
I20260926 19:28:20.924119 11522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:20.924253 11522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e1a33390e04e4c9447bd4ea9092099, State: Initialized, Role: FOLLOWER
I20260926 19:28:20.924782 11522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } }
I20260926 19:28:20.925010 11522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:28:20.925154 11522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:28:20.925364 11522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:20.927706 11522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } }
I20260926 19:28:20.928167 11522 leader_election.cc:302] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e1a33390e04e4c9447bd4ea9092099; no voters:
I20260926 19:28:20.928726 11522 leader_election.cc:288] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:28:20.928896 11527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:28:20.930862 11522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:28:20.933559 11527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [term 1 LEADER]: Becoming Leader. State: Replica: 01e1a33390e04e4c9447bd4ea9092099, State: Running, Role: LEADER
I20260926 19:28:20.934083 11527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } }
I20260926 19:28:20.937044 11529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e1a33390e04e4c9447bd4ea9092099. Latest consensus state: current_term: 1 leader_uuid: "01e1a33390e04e4c9447bd4ea9092099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } } }
I20260926 19:28:20.937410 11529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [sys.catalog]: This master's current role is: LEADER
I20260926 19:28:20.937147 11528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e1a33390e04e4c9447bd4ea9092099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e1a33390e04e4c9447bd4ea9092099" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37569 } } }
I20260926 19:28:20.937565 11528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099 [sys.catalog]: This master's current role is: LEADER
W20260926 19:28:20.943432 11541 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:28:20.943571 11541 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:28:20.943737 11542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:28:20.945854 11542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:28:20.953627 11542 catalog_manager.cc:1347] Generated new cluster ID: 19cf95b4a2b648a5adde07d10c9bc378
I20260926 19:28:20.953708 11542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:28:20.976766 11542 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:28:20.978673 11542 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:28:20.986917 11542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: Generated new TSK 0
I20260926 19:28:20.987941 11542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:28:21.065642 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:28:21.424434 11548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:21.424739 11548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:21.424887 11548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:21.432778 11548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:21.432907 11548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:21.433082 11548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:28:21.444540 11548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:21.445976 11548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:21.447423 11548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:21.460506 11553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:21.460583 11556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:21.460610 11554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:21.461617 11548 server_base.cc:1034] running on GCE node
I20260926 19:28:21.462273 11548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:21.463181 11548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:21.464408 11548 hybrid_clock.cc:648] HybridClock initialized: now 1790450901464355 us; error 44 us; skew 500 ppm
I20260926 19:28:21.464726 11548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:28:21 dist-test-slave-9wb0 krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450901, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:28:21.467854 11548 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:28:21.469386 11548 webserver.cc:460] Webserver started at http://127.1.39.1:35891/ using document root <none> and password file <none>
I20260926 19:28:21.470067 11548 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:21.470227 11548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:21.470527 11548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:21.473569 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "bfceda7bcec546c1a5f00048224fb0f2"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "ecf689f0860d5c759fb04b4dcb817a97"
server_key_iv: "82a0d2770ae3c934a251501a72765944"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.474385 11548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "bfceda7bcec546c1a5f00048224fb0f2"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "ecf689f0860d5c759fb04b4dcb817a97"
server_key_iv: "82a0d2770ae3c934a251501a72765944"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.479430 11548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:28:21.482858 11563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:21.484272 11548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 19:28:21.484449 11548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "bfceda7bcec546c1a5f00048224fb0f2"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "ecf689f0860d5c759fb04b4dcb817a97"
server_key_iv: "82a0d2770ae3c934a251501a72765944"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.484656 11548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:21.498989 11548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:21.503032 11548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:21.503321 11548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:21.504240 11548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:21.505620 11548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:21.505718 11548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:21.505865 11548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:21.505940 11548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:21.539417 11548 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:46049
I20260926 19:28:21.539505 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:46049 every 8 connection(s)
I20260926 19:28:21.540663 11548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:28:21.543962 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11548
I20260926 19:28:21.544222 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:28:21.545039 1180 external_mini_cluster.cc:1428] Setting key c6dca3daac27765fb59a6167e1ab50bd
Sep 26 19:28:21 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450901, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:28:21.563381 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:21.543157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:42557 (local address 127.1.39.62:37569)
0926 19:28:21.543602 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:21.543608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:21.544815 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 19:28:21.547152 (+ 2337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:21.547156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:21.547213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:21.547325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:21.549417 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:21.550127 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:21.552607 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:21.553012 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:21.553110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:21.557610 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:21.557614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:21.557617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:21.557647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:21.560084 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:21.561019 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:21.561022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:21.561025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:21.561144 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:21.561688 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:21.561691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:21.561693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:21.561933 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:21.562063 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:21.562949 (+ 886us) server_negotiation.cc:299] Negotiation successful
0926 19:28:21.563121 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":205,"threads_started":1}
I20260926 19:28:21.565007 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:21.543757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37569 (local address 127.1.39.1:42557)
0926 19:28:21.544627 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:21.544679 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:21.546848 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:21.547577 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:21.547588 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:21.548345 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:21.549178 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:21.549206 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:21.550313 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:21.550321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:21.552437 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:21.552455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:21.553235 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:21.553244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:21.553600 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:21.554326 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:21.554359 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:21.557367 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:21.560233 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:21.560245 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:21.560265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:21.560899 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:21.561252 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:21.561260 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:21.561267 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:21.561573 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:21.562064 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:21.562075 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:21.562541 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 19:28:21.562870 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 19:28:21.563477 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":361,"threads_started":1}
I20260926 19:28:21.566701 11676 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37569
I20260926 19:28:21.567016 11676 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:21.567759 11676 heartbeater.cc:511] Master 127.1.39.62:37569 requested a full tablet report, sending...
I20260926 19:28:21.570248 11487 ts_manager.cc:194] Registered new tserver with Master: bfceda7bcec546c1a5f00048224fb0f2 (127.1.39.1:46049)
I20260926 19:28:21.572427 11487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:42557
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:28:21.605566 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:28:21.929137 11682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:21.929497 11682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:21.929651 11682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:21.937206 11682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:21.937383 11682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:21.937562 11682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:28:21.947568 11682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:21.948864 11682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:21.950237 11682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:21.964017 11687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:21.963938 11688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:21.964017 11690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:21.965197 11682 server_base.cc:1034] running on GCE node
I20260926 19:28:21.965886 11682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:21.966902 11682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:21.968138 11682 hybrid_clock.cc:648] HybridClock initialized: now 1790450901968068 us; error 67 us; skew 500 ppm
I20260926 19:28:21.968463 11682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:28:21 dist-test-slave-9wb0 krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450901, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:28:21.971627 11682 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:28:21.973129 11682 webserver.cc:460] Webserver started at http://127.1.39.2:36629/ using document root <none> and password file <none>
I20260926 19:28:21.973853 11682 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:21.974002 11682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:21.974300 11682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:21.977248 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "ef1bbb98e9df471491107208c56daf97"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "376ea113672eef3b04785872d6d42ba7"
server_key_iv: "7a0be12dc08af9c4cebfffbdf1e64005"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.978107 11682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "ef1bbb98e9df471491107208c56daf97"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "376ea113672eef3b04785872d6d42ba7"
server_key_iv: "7a0be12dc08af9c4cebfffbdf1e64005"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.983047 11682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:28:21.986321 11697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:21.987713 11682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:28:21.987918 11682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "ef1bbb98e9df471491107208c56daf97"
format_stamp: "Formatted at 2026-09-26 19:28:21 on dist-test-slave-9wb0"
server_key: "376ea113672eef3b04785872d6d42ba7"
server_key_iv: "7a0be12dc08af9c4cebfffbdf1e64005"
server_key_version: "encryptionkey@0"
I20260926 19:28:21.988104 11682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:22.001755 11682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:22.005944 11682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:22.006245 11682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:22.007191 11682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:22.008522 11682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:22.008594 11682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:22.008737 11682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:22.008791 11682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:22.043483 11682 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:43921
I20260926 19:28:22.043583 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:43921 every 8 connection(s)
I20260926 19:28:22.044775 11682 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:28:22.049018 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11682
I20260926 19:28:22.049237 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:28:22.049800 1180 external_mini_cluster.cc:1428] Setting key 1d448b394d04c5112e527258fcfe018d
Sep 26 19:28:22 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450901, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:28:22.068594 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.047716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:34313 (local address 127.1.39.62:37569)
0926 19:28:22.047883 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.047888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.049160 (+ 1272us) server_negotiation.cc:407] Connection header received
0926 19:28:22.051460 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.051464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.051588 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.051698 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:22.053731 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.054423 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.056796 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.057213 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.057362 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.061436 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:22.061444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:22.061451 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:22.061503 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:22.065047 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.065913 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.065921 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.065928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.066087 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.066616 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.066623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.066629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.067008 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:22.067152 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.067889 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.068143 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:28:22.069447 11812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.048167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37569 (local address 127.1.39.2:34313)
0926 19:28:22.048965 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.049033 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.051199 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.051965 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.051977 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.052712 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:22.053556 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.053582 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.054575 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.054584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.056651 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.056667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.057449 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.057458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.057781 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.058547 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:22.058581 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:22.061187 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:22.065184 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:22.065190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:22.065201 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:22.065772 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:22.066250 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:22.066254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:22.066257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:22.066487 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:22.067138 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:22.067144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:22.067501 (+ 357us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.067744 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.068146 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":282,"threads_started":1}
I20260926 19:28:22.070946 11810 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37569
I20260926 19:28:22.071280 11810 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:22.072036 11810 heartbeater.cc:511] Master 127.1.39.62:37569 requested a full tablet report, sending...
I20260926 19:28:22.074016 11487 ts_manager.cc:194] Registered new tserver with Master: ef1bbb98e9df471491107208c56daf97 (127.1.39.2:43921)
I20260926 19:28:22.075223 11487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:34313
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:28:22.108912 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:28:22.443241 11815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:28:22.443711 11815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:28:22.443934 11815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:28:22.451565 11815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:28:22.451674 11815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:28:22.451856 11815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:28:22.461855 11815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:28:22.463320 11815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:28:22.464704 11815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:28:22.477864 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:22.478057 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:28:22.477803 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:28:22.482838 11815 server_base.cc:1034] running on GCE node
I20260926 19:28:22.483512 11815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:28:22.488620 11815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:28:22.489857 11815 hybrid_clock.cc:648] HybridClock initialized: now 1790450902489793 us; error 52 us; skew 500 ppm
I20260926 19:28:22.490214 11815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:28:22 dist-test-slave-9wb0 krb5kdc[11442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:28:22.494567 11815 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:28:22.502110 11815 webserver.cc:460] Webserver started at http://127.1.39.3:41097/ using document root <none> and password file <none>
I20260926 19:28:22.502978 11815 fs_manager.cc:359] Metadata directory not provided
I20260926 19:28:22.503144 11815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:28:22.503465 11815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:28:22.506537 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "eacb79acd1ae4d9db9db4e09cfd7c766"
format_stamp: "Formatted at 2026-09-26 19:28:22 on dist-test-slave-9wb0"
server_key: "d385286ded132d4ba738c75ba1f8904a"
server_key_iv: "c4488cf00ca7861865eefb71998cd0e7"
server_key_version: "encryptionkey@0"
I20260926 19:28:22.507328 11815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "eacb79acd1ae4d9db9db4e09cfd7c766"
format_stamp: "Formatted at 2026-09-26 19:28:22 on dist-test-slave-9wb0"
server_key: "d385286ded132d4ba738c75ba1f8904a"
server_key_iv: "c4488cf00ca7861865eefb71998cd0e7"
server_key_version: "encryptionkey@0"
I20260926 19:28:22.512748 11815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 19:28:22.516299 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:22.517835 11815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:28:22.518049 11815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "eacb79acd1ae4d9db9db4e09cfd7c766"
format_stamp: "Formatted at 2026-09-26 19:28:22 on dist-test-slave-9wb0"
server_key: "d385286ded132d4ba738c75ba1f8904a"
server_key_iv: "c4488cf00ca7861865eefb71998cd0e7"
server_key_version: "encryptionkey@0"
I20260926 19:28:22.518276 11815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:28:22.543246 11815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:28:22.548213 11815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:28:22.548501 11815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:28:22.549484 11815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:28:22.550773 11815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:28:22.550845 11815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:22.550925 11815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:28:22.550971 11815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:28:22.575476 11676 heartbeater.cc:503] Master 127.1.39.62:37569 was elected leader, sending a full tablet report...
I20260926 19:28:22.583853 11815 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:40065
I20260926 19:28:22.583855 11942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:40065 every 8 connection(s)
I20260926 19:28:22.585158 11815 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:28:22.594370 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 11815
I20260926 19:28:22.594573 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:28:22.594970 1180 external_mini_cluster.cc:1428] Setting key f9af0247c73907618d12ed718bd2ba60
Sep 26 19:28:22 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450902, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:28:22.605841 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.587743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:34719 (local address 127.1.39.62:37569)
0926 19:28:22.588179 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.588184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.589541 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 19:28:22.591368 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.591374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.591465 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.591649 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:22.593182 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.594021 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.595834 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.596412 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.596560 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.600142 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:22.600147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:22.600150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:22.600184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:22.602913 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.603592 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.603596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.603598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.603719 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.604126 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.604131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.604133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.604514 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:22.604646 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.605288 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.605539 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":232,"threads_started":1}
I20260926 19:28:22.606806 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.588321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37569 (local address 127.1.39.3:34719)
0926 19:28:22.589350 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.589410 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.591105 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.591873 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.591881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.592372 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:28:22.593018 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.593035 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.594169 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.594173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.595709 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.595721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.596630 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.596635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.596928 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.597686 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:28:22.597708 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:28:22.599946 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:28:22.603043 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:22.603049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:22.603065 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:22.603469 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:22.603824 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:28:22.603827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:28:22.603829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:28:22.603997 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:28:22.604637 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:28:22.604643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:28:22.604947 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.605178 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.605629 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":457,"threads_started":1}
I20260926 19:28:22.608264 11943 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37569
I20260926 19:28:22.608556 11943 heartbeater.cc:464] Registering TS with master...
I20260926 19:28:22.609408 11943 heartbeater.cc:511] Master 127.1.39.62:37569 requested a full tablet report, sending...
I20260926 19:28:22.610834 11487 ts_manager.cc:194] Registered new tserver with Master: eacb79acd1ae4d9db9db4e09cfd7c766 (127.1.39.3:40065)
I20260926 19:28:22.611572 11487 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:34719
I20260926 19:28:22.623330 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:28:22.640503 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.627919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.1.39.62:37569)
0926 19:28:22.628146 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.628155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.628550 (+ 395us) server_negotiation.cc:407] Connection header received
0926 19:28:22.628673 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.628676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.628742 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.628822 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:22.630379 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.631226 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.632588 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.633036 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.633134 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.635541 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:22.635547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:22.635550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:22.635590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:22.638038 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.638653 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.638658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.638660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.638790 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:22.639100 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:22.639104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:22.639106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:22.639419 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:22.639624 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.639936 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.640148 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:28:22.644690 11487 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:28:22.651975 11487 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:28:22.678224 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.665178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:40065 (local address 127.0.0.1:35928)
0926 19:28:22.665749 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.665768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.665888 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.667205 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.667213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.667249 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:28:22.667781 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.667796 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.670659 (+ 2863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.670667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.674168 (+ 3501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.674192 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.676769 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.676778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.677045 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.677103 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.677197 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.677486 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":56,"thread_start_us":295,"threads_started":1}
I20260926 19:28:22.678890 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.664174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.1.39.2:43921)
0926 19:28:22.664783 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.664795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.664981 (+ 186us) server_negotiation.cc:407] Connection header received
0926 19:28:22.665454 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.665468 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.665770 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.665982 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:28:22.667282 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.669235 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.673734 (+ 4499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.675889 (+ 2155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.676073 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.676361 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.678209 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.678471 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":353,"threads_started":1}
I20260926 19:28:22.679617 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.665576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.1.39.3:40065)
0926 19:28:22.666476 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.666484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.666515 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:28:22.666606 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.666615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.666935 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.668030 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:28:22.668397 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.670474 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.674459 (+ 3985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.676469 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.678776 (+ 2307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.679038 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.679108 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.679284 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":597,"threads_started":1}
I20260926 19:28:22.679947 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.663988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:43921 (local address 127.0.0.1:42762)
0926 19:28:22.664833 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.664871 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.665088 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.666294 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.666302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.666337 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:28:22.666959 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.666980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.669485 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.669494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.673464 (+ 3970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.673490 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.676526 (+ 3036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.676536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.676889 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.676962 (+ 73us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.677139 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.677485 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":82,"thread_start_us":415,"threads_started":1}
I20260926 19:28:22.683983 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 84be4adab3af4caaaf25f98167e01028 (DEFAULT_TABLE table=test-table [id=c6ba45f3d2de474094ca299e069ca13b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:28:22.684024 11745 tablet_service.cc:1467] Processing CreateTablet for tablet 84be4adab3af4caaaf25f98167e01028 (DEFAULT_TABLE table=test-table [id=c6ba45f3d2de474094ca299e069ca13b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:28:22.685400 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be4adab3af4caaaf25f98167e01028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:28:22.685411 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be4adab3af4caaaf25f98167e01028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:28:22.687827 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.666520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46049 (local address 127.0.0.1:55622)
0926 19:28:22.678815 (+ 12295us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.678836 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.678965 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.680553 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.680557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.680583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:28:22.680920 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.680929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.682849 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.682854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.684833 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.684848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.687000 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.687008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.687192 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.687232 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.687342 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.687484 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12160,"thread_start_us":12512,"threads_started":1}
I20260926 19:28:22.687951 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.666228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.1.39.1:46049)
0926 19:28:22.679942 (+ 13714us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.679951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.679982 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:28:22.680102 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.680111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.680354 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.680569 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:28:22.681104 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.682683 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.685002 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.686502 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.686632 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.686855 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.687406 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.687578 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13528,"thread_start_us":13377,"threads_started":1}
I20260926 19:28:22.692968 11611 tablet_service.cc:1467] Processing CreateTablet for tablet 84be4adab3af4caaaf25f98167e01028 (DEFAULT_TABLE table=test-table [id=c6ba45f3d2de474094ca299e069ca13b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:28:22.694252 11611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84be4adab3af4caaaf25f98167e01028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:28:22.698480 11962 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Bootstrap starting.
I20260926 19:28:22.702462 11962 tablet_bootstrap.cc:654] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:22.703919 11962 log.cc:824] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:22.703976 11963 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Bootstrap starting.
I20260926 19:28:22.707217 11963 tablet_bootstrap.cc:654] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:22.707686 11962 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: No bootstrap required, opened a new log
I20260926 19:28:22.708010 11962 ts_tablet_manager.cc:1397] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 19:28:22.708477 11965 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Bootstrap starting.
I20260926 19:28:22.708413 11963 log.cc:824] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:22.712065 11965 tablet_bootstrap.cc:654] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Neither blocks nor log segments found. Creating new log.
I20260926 19:28:22.712934 11963 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: No bootstrap required, opened a new log
I20260926 19:28:22.713224 11963 ts_tablet_manager.cc:1397] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260926 19:28:22.713348 11965 log.cc:824] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:28:22.717190 11965 tablet_bootstrap.cc:492] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: No bootstrap required, opened a new log
I20260926 19:28:22.717043 11962 raft_consensus.cc:348] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.717525 11965 ts_tablet_manager.cc:1397] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:28:22.717523 11962 raft_consensus.cc:374] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:22.717634 11962 raft_consensus.cc:729] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef1bbb98e9df471491107208c56daf97, State: Initialized, Role: FOLLOWER
I20260926 19:28:22.718343 11962 consensus_queue.cc:260] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.719774 11810 heartbeater.cc:503] Master 127.1.39.62:37569 was elected leader, sending a full tablet report...
I20260926 19:28:22.720306 11962 ts_tablet_manager.cc:1428] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Time spent starting tablet: real 0.012s user 0.001s sys 0.010s
I20260926 19:28:22.725143 11963 raft_consensus.cc:348] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.725654 11963 raft_consensus.cc:374] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:22.725750 11963 raft_consensus.cc:729] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eacb79acd1ae4d9db9db4e09cfd7c766, State: Initialized, Role: FOLLOWER
I20260926 19:28:22.726364 11963 consensus_queue.cc:260] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.727414 11943 heartbeater.cc:503] Master 127.1.39.62:37569 was elected leader, sending a full tablet report...
I20260926 19:28:22.728179 11963 ts_tablet_manager.cc:1428] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260926 19:28:22.729604 11965 raft_consensus.cc:348] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.730049 11965 raft_consensus.cc:374] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:28:22.730248 11965 raft_consensus.cc:729] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfceda7bcec546c1a5f00048224fb0f2, State: Initialized, Role: FOLLOWER
I20260926 19:28:22.730832 11965 consensus_queue.cc:260] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.732705 11965 ts_tablet_manager.cc:1428] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260926 19:28:22.793030 11677 tablet.cc:2369] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:28:22.796734 11811 tablet.cc:2369] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:28:22.836771 11944 tablet.cc:2369] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:28:22.895056 11969 raft_consensus.cc:482] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:28:22.895432 11969 raft_consensus.cc:504] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.897253 11969 leader_election.cc:288] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef1bbb98e9df471491107208c56daf97 (127.1.39.2:43921), bfceda7bcec546c1a5f00048224fb0f2 (127.1.39.1:46049)
I20260926 19:28:22.903702 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.897512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:43921 (local address 127.1.39.3:53071)
0926 19:28:22.897721 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.897744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.897873 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.898582 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.898588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.898630 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:28:22.898845 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.898855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.900156 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.900164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.901954 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.901966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.903175 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.903179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.903300 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.903322 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.903395 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.903478 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1}
I20260926 19:28:22.903884 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.897685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:53071 (local address 127.1.39.2:43921)
0926 19:28:22.897862 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.897867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.897889 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:28:22.898087 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.898097 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.898212 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.898330 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:28:22.899030 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.899943 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.902121 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.903013 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.903183 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.903294 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.903453 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.903598 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:28:22.904912 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be4adab3af4caaaf25f98167e01028" candidate_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1bbb98e9df471491107208c56daf97" is_pre_election: true
I20260926 19:28:22.905273 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.898187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:51369 (local address 127.1.39.1:46049)
0926 19:28:22.898353 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.898359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.898993 (+ 634us) server_negotiation.cc:407] Connection header received
0926 19:28:22.899242 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.899248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.899335 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.899462 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:28:22.900063 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.901179 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.903125 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.904373 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.904531 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.904608 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.904951 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.905065 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:28:22.905441 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.897887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46049 (local address 127.1.39.3:51369)
0926 19:28:22.898892 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 19:28:22.898909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:28:22.899082 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:28:22.899491 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:28:22.899498 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:28:22.899524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:28:22.899892 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.899903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.901386 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.901392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.902971 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:28:22.902982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.904588 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.904594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:28:22.904764 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.904804 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:28:22.904911 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 19:28:22.905036 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":5,"thread_start_us":610,"threads_started":1}
I20260926 19:28:22.905503 11765 raft_consensus.cc:2454] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eacb79acd1ae4d9db9db4e09cfd7c766 in term 0.
I20260926 19:28:22.906226 11834 leader_election.cc:302] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eacb79acd1ae4d9db9db4e09cfd7c766, ef1bbb98e9df471491107208c56daf97; no voters:
I20260926 19:28:22.906606 11969 raft_consensus.cc:2781] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:28:22.906441 11631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be4adab3af4caaaf25f98167e01028" candidate_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfceda7bcec546c1a5f00048224fb0f2" is_pre_election: true
I20260926 19:28:22.906757 11969 raft_consensus.cc:482] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:28:22.906869 11969 raft_consensus.cc:3037] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:22.907008 11631 raft_consensus.cc:2454] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eacb79acd1ae4d9db9db4e09cfd7c766 in term 0.
I20260926 19:28:22.909134 11969 raft_consensus.cc:504] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.909688 11969 leader_election.cc:288] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [CANDIDATE]: Term 1 election: Requested vote from peers ef1bbb98e9df471491107208c56daf97 (127.1.39.2:43921), bfceda7bcec546c1a5f00048224fb0f2 (127.1.39.1:46049)
I20260926 19:28:22.910135 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be4adab3af4caaaf25f98167e01028" candidate_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef1bbb98e9df471491107208c56daf97"
I20260926 19:28:22.910166 11631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84be4adab3af4caaaf25f98167e01028" candidate_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfceda7bcec546c1a5f00048224fb0f2"
I20260926 19:28:22.910331 11765 raft_consensus.cc:3037] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:22.910341 11631 raft_consensus.cc:3037] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:28:22.912402 11631 raft_consensus.cc:2454] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eacb79acd1ae4d9db9db4e09cfd7c766 in term 1.
I20260926 19:28:22.912423 11765 raft_consensus.cc:2454] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eacb79acd1ae4d9db9db4e09cfd7c766 in term 1.
I20260926 19:28:22.912756 11834 leader_election.cc:302] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfceda7bcec546c1a5f00048224fb0f2, eacb79acd1ae4d9db9db4e09cfd7c766; no voters:
I20260926 19:28:22.913022 11969 raft_consensus.cc:2781] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:28:22.913508 11969 raft_consensus.cc:686] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 1 LEADER]: Becoming Leader. State: Replica: eacb79acd1ae4d9db9db4e09cfd7c766, State: Running, Role: LEADER
I20260926 19:28:22.913942 11969 consensus_queue.cc:237] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } }
I20260926 19:28:22.918474 11487 catalog_manager.cc:5495] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 reported cstate change: term changed from 0 to 1, leader changed from <none> to eacb79acd1ae4d9db9db4e09cfd7c766 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eacb79acd1ae4d9db9db4e09cfd7c766" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 40065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 } health_report { overall_health: UNKNOWN } } }
I20260926 19:28:22.983418 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:22.978856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.1.39.3:40065)
0926 19:28:22.979032 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:22.979036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:22.979052 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:28:22.979141 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:22.979144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:22.979216 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:22.979338 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:28:22.979667 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.980362 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.981640 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:22.982034 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:22.982127 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:22.982515 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:28:22.982996 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:28:22.983097 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:22.983172 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 19:28:22.983229 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:28:22.991561 11631 raft_consensus.cc:1261] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 1 FOLLOWER]: Refusing update from remote peer eacb79acd1ae4d9db9db4e09cfd7c766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:28:22.991658 11765 raft_consensus.cc:1261] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 1 FOLLOWER]: Refusing update from remote peer eacb79acd1ae4d9db9db4e09cfd7c766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:28:22.992537 11972 consensus_queue.cc:1035] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef1bbb98e9df471491107208c56daf97" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:28:22.992897 11969 consensus_queue.cc:1035] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfceda7bcec546c1a5f00048224fb0f2" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:28:23.004312 11975 mvcc.cc:205] Tried to move back new op lower bound from 7333686898644381696 to 7333686898339295232. Current Snapshot: MvccSnapshot[applied={T|T < 7333686898644381696}]
I20260926 19:28:23.016109 11486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:28:23.016450 11486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:28:23.020264 11486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e1a33390e04e4c9447bd4ea9092099: Sending DeleteTablet for 3 replicas of tablet 84be4adab3af4caaaf25f98167e01028
I20260926 19:28:23.021615 11611 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be4adab3af4caaaf25f98167e01028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:28:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:55622
I20260926 19:28:23.021813 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be4adab3af4caaaf25f98167e01028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:28:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:35928
I20260926 19:28:23.022207 11745 tablet_service.cc:1514] Processing DeleteTablet for tablet 84be4adab3af4caaaf25f98167e01028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:28:23 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:42762
I20260926 19:28:23.022776 11985 tablet_replica.cc:330] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: stopping tablet replica
I20260926 19:28:23.023372 11985 raft_consensus.cc:2229] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:28:23.023738 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11548
I20260926 19:28:23.024154 11985 raft_consensus.cc:2258] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:28:23.025856 11984 tablet_replica.cc:330] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: stopping tablet replica
I20260926 19:28:23.026218 11984 raft_consensus.cc:2229] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:28:23.026650 11984 raft_consensus.cc:2258] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:28:23.028560 11984 ts_tablet_manager.cc:1905] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:28:23.035166 11984 ts_tablet_manager.cc:1918] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:28:23.035301 11985 ts_tablet_manager.cc:1905] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:28:23.035459 11984 log.cc:1196] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/84be4adab3af4caaaf25f98167e01028
I20260926 19:28:23.036088 11984 ts_tablet_manager.cc:1939] T 84be4adab3af4caaaf25f98167e01028 P bfceda7bcec546c1a5f00048224fb0f2: Deleting consensus metadata
I20260926 19:28:23.038064 11474 catalog_manager.cc:4841] TS bfceda7bcec546c1a5f00048224fb0f2 (127.1.39.1:46049): tablet 84be4adab3af4caaaf25f98167e01028 (table test-table [id=c6ba45f3d2de474094ca299e069ca13b]) successfully deleted
I20260926 19:28:23.038335 11986 tablet_replica.cc:330] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: stopping tablet replica
I20260926 19:28:23.038806 11986 raft_consensus.cc:2229] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:28:23.039330 11986 raft_consensus.cc:2258] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:28:23.041466 11986 ts_tablet_manager.cc:1905] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:28:23.044903 11985 ts_tablet_manager.cc:1918] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:28:23.045071 11985 log.cc:1196] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/84be4adab3af4caaaf25f98167e01028
I20260926 19:28:23.045708 11985 ts_tablet_manager.cc:1939] T 84be4adab3af4caaaf25f98167e01028 P eacb79acd1ae4d9db9db4e09cfd7c766: Deleting consensus metadata
I20260926 19:28:23.047495 11474 catalog_manager.cc:4841] TS eacb79acd1ae4d9db9db4e09cfd7c766 (127.1.39.3:40065): tablet 84be4adab3af4caaaf25f98167e01028 (table test-table [id=c6ba45f3d2de474094ca299e069ca13b]) successfully deleted
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:28:23.054153 11986 ts_tablet_manager.cc:1918] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:28:23.054363 11986 log.cc:1196] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/84be4adab3af4caaaf25f98167e01028
I20260926 19:28:23.054988 11986 ts_tablet_manager.cc:1939] T 84be4adab3af4caaaf25f98167e01028 P ef1bbb98e9df471491107208c56daf97: Deleting consensus metadata
I20260926 19:28:23.056914 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:23.025057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.1.39.1:46049)
0926 19:28:23.025244 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:23.025254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:23.037827 (+ 12573us) server_negotiation.cc:407] Connection header received
0926 19:28:23.038068 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:23.038074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:23.038155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:23.038231 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:23.040482 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.041554 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.043380 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.044001 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.044091 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:23.049611 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:23.049627 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:23.049655 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:23.049699 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:23.053418 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.054261 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.054329 (+ 68us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.054332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.054547 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.055090 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.055096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.055100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.055537 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:23.055798 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:23.056353 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 19:28:23.056563 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 19:28:23.056900 11474 catalog_manager.cc:4841] TS ef1bbb98e9df471491107208c56daf97 (127.1.39.2:43921): tablet 84be4adab3af4caaaf25f98167e01028 (table test-table [id=c6ba45f3d2de474094ca299e069ca13b]) successfully deleted
I20260926 19:28:23.058288 11671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:23.146021 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11548
I20260926 19:28:23.164362 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11682
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:28:23.194196 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:23.165164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.1.39.2:43921)
0926 19:28:23.165398 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:23.165402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:23.165419 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:28:23.165594 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:23.165597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:23.165693 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:23.165777 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:23.167367 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.168136 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.169997 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.170411 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.170534 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:23.187768 (+ 17234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:23.187786 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:23.187804 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:23.187845 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:23.190953 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.191808 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.191817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.191819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.191947 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.192460 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.192464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.192467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.192877 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:23.193192 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:23.193606 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 19:28:23.193840 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:28:23.194882 11805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:23.285311 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11682
I20260926 19:28:23.302570 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11815
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:28:23.317991 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:28:23.303358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.1.39.3:40065)
0926 19:28:23.303548 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:28:23.303556 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:28:23.303646 (+ 90us) server_negotiation.cc:407] Connection header received
0926 19:28:23.303852 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:28:23.303858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:28:23.303931 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:28:23.304064 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:28:23.305262 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.306079 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.307276 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:28:23.307729 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:28:23.307866 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:28:23.312057 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:28:23.312069 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:28:23.312087 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:28:23.312127 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:28:23.315216 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.315870 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.315878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.315881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.316029 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:28:23.316411 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:28:23.316416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:28:23.316420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:28:23.316779 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:28:23.317157 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 19:28:23.317394 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 19:28:23.317606 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:28:23.318640 11938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:23.438901 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11815
I20260926 19:28:23.461448 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 11455
I20260926 19:28:23.462527 11517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:28:23.571164 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 11455
2026-09-26T19:28:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4469 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11994](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11994](info): set up 2 sockets
Sep 26 19:28:23 dist-test-slave-9wb0 krb5kdc[11994](info): commencing operation
krb5kdc: starting...
W20260926 19:28:24.690089 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:28:24 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:28:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangeradmin_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangeradmin_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangerlookup_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangerlookup_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
I20260926 19:28:25.044107 1180 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksjShy7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/postgres -l logfile start
2026-09-26 19:28:27.658 UTC [12024] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 19:28:27.659 UTC [12024] LOG: listening on IPv4 address "127.1.39.20", port 35875
2026-09-26 19:28:27.662 UTC [12024] LOG: listening on Unix socket "/tmp/.s.PGSQL.35875"
2026-09-26 19:28:27.684 UTC [12027] LOG: database system was shut down at 2026-09-26 19:28:26 UTC
2026-09-26 19:28:27.688 UTC [12024] LOG: database system is ready to accept connections
I20260926 19:28:28.730010 1180 mini_postgres.cc:96] Postgres bound to 35875
2026-09-26 19:28:28.750 UTC [12035] FATAL: database "slave" does not exist
127.1.39.20:35875 - accepting connections
I20260926 19:28:28.750949 1180 mini_ranger.cc:161] Starting Ranger...
I20260926 19:28:28.778684 1180 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 19:28:29.292012 1180 mini_ranger.cc:87] Created ranger Postgres database
I20260926 19:28:29.292155 1180 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin
2026-09-26 19:28:29,594 [I] DB FLAVOR :POSTGRES
2026-09-26 19:28:29,594 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:28:29,594 [I] Checking connection..
2026-09-26 19:28:29,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 19:28:29,848 [I] Checking connection passed.
2026-09-26 19:28:29,848 [I] --------- Verifying version history table ---------
2026-09-26 19:28:29,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:28:30,089 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:28:30,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:28:30,339 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:28:30,340 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 19:28:30,340 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 19:28:30.587 UTC [12099] WARNING: there is no transaction in progress
2026-09-26 19:28:30,594 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 19:28:30,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:28:30,859 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 19:28:30,859 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:28:30,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 19:28:31,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 19:28:31,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9wb0.c.gcp-upstream.internal','N') ;"
2026-09-26 19:28:31,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:28:31,874 [I] Table x_portal_user does not exist in database ranger
2026-09-26 19:28:31,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:28:32,127 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 19:28:32,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:28:32,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:28:32,664 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 19:28:32,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 19:28:33.648 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.664 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.680 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.695 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.711 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.730 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.784 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.792 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.807 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.817 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.828 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.842 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.851 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.858 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.864 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:33.871 UTC [12207] WARNING: there is no transaction in progress
2026-09-26 19:28:34,511 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 19:28:34,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 19:28:34,763 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 19:28:34,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:28:35,032 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 19:28:35,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:28:35,290 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 19:28:35,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:28:35,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:28:35,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9wb0.c.gcp-upstream.internal';"
2026-09-26 19:28:36,026 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 19:28:36,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 19:28:36,279 [I] DB_PATCHES have already been applied
I20260926 19:28:36.285377 1180 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksjShy7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 19:28:36.285463 1180 mini_ranger.cc:193] Using host: 127.1.39.20
I20260926 19:28:36.303088 1180 mini_ranger.cc:227] Ranger admin URL: http://127.1.39.20:60283
Sep 26, 2026 7:28:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:28:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:28:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:28:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:28:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:28:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:28:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 19:28:37.468665 1180 mini_ranger.cc:160] Time spent starting Ranger: real 8.718s user 0.004s sys 0.041s
Sep 26, 2026 7:28:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:28:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:28:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:28:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:28:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 19:28:58.530746 1180 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangerkms_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/rangerkms_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.39.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.39.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 19:28:58 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450938, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 19:28:58.723150 1180 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 19:28:58.746764 1180 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 19:28:59.747385 1180 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 19:28:59.747568 1180 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms
I20260926 19:28:59.747700 1180 mini_ranger_kms.cc:227] Using postgres at 127.1.39.20:35875
2026-09-26 19:29:00,247 [I] DB FLAVOR :POSTGRES
2026-09-26 19:29:00,247 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:29:00,247 [I] Checking connection
2026-09-26 19:29:00,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 19:29:00,518 [I] connection success
2026-09-26 19:29:00,518 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 19:29:00,518 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 19:29:00,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 19:29:00,785 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 19:29:00,785 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:29:00,785 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 19:29:00,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:35875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 19:29:01,075 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 19:29:01.554718 1180 mini_ranger_kms.cc:310] Created kms service
I20260926 19:29:01.752390 1180 mini_ranger_kms.cc:326] Created kudu user
I20260926 19:29:01.800930 1180 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 19:29:01 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450938, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:02.227422 1180 mini_ranger_kms.cc:384] Added ranger policy
I20260926 19:29:02.227753 1180 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 19:29:02.227821 1180 mini_ranger_kms.cc:240] Using host: 127.1.39.20
I20260926 19:29:02.252883 1180 mini_ranger_kms.cc:276] Ranger KMS PID: 12387
I20260926 19:29:02.253077 1180 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.39.20:49207
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:29:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 19:29:04.595950 1180 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.873s user 0.118s sys 0.405s
I20260926 19:29:04.596241 1180 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 19:29:04.596477 1180 mini_ranger_kms.cc:401] 127.1.39.20:49207/kms/v1/keys
Sep 26 19:29:04 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790450944, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 19:29:05 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450944, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
Sep 26 19:29:05 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450944, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
Sep 26 19:29:07 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:29:07.804602 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42593
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42593
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:49207/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:29:08.188220 12438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:29:08.188591 12438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:29:08.188741 12438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:29:08.196496 12438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:29:08.196672 12438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:29:08.196771 12438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:29:08.196871 12438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:29:08.196950 12438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:29:08.207523 12438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.39.20:49207/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42593
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:29:08.208875 12438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:29:08.210284 12438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:29:08.224112 12446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:08.224140 12444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:08.224686 12438 server_base.cc:1034] running on GCE node
W20260926 19:29:08.224898 12443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:08.225780 12438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:29:08.227248 12438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:29:08.228477 12438 hybrid_clock.cc:648] HybridClock initialized: now 1790450948228421 us; error 49 us; skew 500 ppm
I20260926 19:29:08.228820 12438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:29:08 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450948, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:29:08.232236 12438 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:29:08.233863 12438 webserver.cc:460] Webserver started at http://127.1.39.62:37651/ using document root <none> and password file <none>
I20260926 19:29:08.234515 12438 fs_manager.cc:359] Metadata directory not provided
I20260926 19:29:08.234637 12438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:29:08.234946 12438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:29:08 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450948, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:08.297719 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "ffc14654987c4ca3b93d1f06586a1522"
format_stamp: "Formatted at 2026-09-26 19:29:08 on dist-test-slave-9wb0"
server_key: "bee53e7b55b3de9a032bfa363519f628"
server_key_iv: "a1bb7c45ac3a2dbe421dd7ece4bcbaee"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:08.298563 12438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "ffc14654987c4ca3b93d1f06586a1522"
format_stamp: "Formatted at 2026-09-26 19:29:08 on dist-test-slave-9wb0"
server_key: "bee53e7b55b3de9a032bfa363519f628"
server_key_iv: "a1bb7c45ac3a2dbe421dd7ece4bcbaee"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:08.304240 12438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:29:08.320693 12453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:08.322288 12438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:29:08.322486 12438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "ffc14654987c4ca3b93d1f06586a1522"
format_stamp: "Formatted at 2026-09-26 19:29:08 on dist-test-slave-9wb0"
server_key: "bee53e7b55b3de9a032bfa363519f628"
server_key_iv: "a1bb7c45ac3a2dbe421dd7ece4bcbaee"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:08.322712 12438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:29:08.345227 12438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:29:08.349435 12438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:29:08.349745 12438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:29:08.371479 12438 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42593
I20260926 19:29:08.371487 12504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42593 every 8 connection(s)
I20260926 19:29:08.372924 12438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:29:08.374399 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 12438
I20260926 19:29:08.374614 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:29:08.375427 1180 external_mini_cluster.cc:1428] Setting key 94cf14517f99f4b02901d01c1f33dc02
I20260926 19:29:08.379828 12505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:29:08.396477 12505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: Bootstrap starting.
I20260926 19:29:08.400554 12505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: Neither blocks nor log segments found. Creating new log.
I20260926 19:29:08.402004 12505 log.cc:824] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:29:08 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:29:08.405519 12505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: No bootstrap required, opened a new log
I20260926 19:29:08.414244 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:08.377734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.1.39.62:42593)
0926 19:29:08.389917 (+ 12183us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:08.389934 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:08.390007 (+ 73us) server_negotiation.cc:407] Connection header received
0926 19:29:08.392511 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:08.392545 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:08.393106 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:08.393586 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:08.395198 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:08.397245 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:08.399537 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:08.400512 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:08.400685 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:08.405333 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:08.405344 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:08.405362 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:08.405406 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:08.409357 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:08.410202 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:08.410214 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:08.410223 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:08.410439 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:08.410896 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:08.410901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:08.410904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:08.411625 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:08.411918 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:08.412394 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 19:29:08.412812 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11916,"thread_start_us":11686,"threads_started":1}
I20260926 19:29:08.414573 12505 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } }
I20260926 19:29:08.415402 12505 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:29:08.415534 12505 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc14654987c4ca3b93d1f06586a1522, State: Initialized, Role: FOLLOWER
I20260926 19:29:08.416273 12505 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } }
I20260926 19:29:08.416590 12505 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:29:08.416751 12505 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:29:08.416946 12505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:29:08.419449 12505 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } }
I20260926 19:29:08.420085 12505 leader_election.cc:302] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc14654987c4ca3b93d1f06586a1522; no voters:
I20260926 19:29:08.420804 12505 leader_election.cc:288] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:29:08.420912 12510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:29:08.423986 12505 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:29:08.425951 12510 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [term 1 LEADER]: Becoming Leader. State: Replica: ffc14654987c4ca3b93d1f06586a1522, State: Running, Role: LEADER
I20260926 19:29:08.427395 12510 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } }
I20260926 19:29:08.430814 12511 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc14654987c4ca3b93d1f06586a1522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } } }
I20260926 19:29:08.431318 12511 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [sys.catalog]: This master's current role is: LEADER
I20260926 19:29:08.430902 12512 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc14654987c4ca3b93d1f06586a1522. Latest consensus state: current_term: 1 leader_uuid: "ffc14654987c4ca3b93d1f06586a1522" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc14654987c4ca3b93d1f06586a1522" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42593 } } }
I20260926 19:29:08.431545 12512 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522 [sys.catalog]: This master's current role is: LEADER
I20260926 19:29:08.432626 12516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:29:08.446650 12516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:29:08.471830 12516 catalog_manager.cc:1347] Generated new cluster ID: a0d393301e554ac9928495e0f63a6add
I20260926 19:29:08.471957 12516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:29:08.490444 12516 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:29:08.492358 12516 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:29:08.514995 12516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: Generated new TSK 0
I20260926 19:29:08.516223 12516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:29:08.519516 12505 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:29:09.374963 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:49207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:29:09 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 19:29:09.818225 12550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:29:09.818660 12550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:29:09.818885 12550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:29:09.831502 12550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:29:09.831619 12550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:29:09.831794 12550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:29:09.844336 12550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.39.20:49207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:29:09.846309 12550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:29:09.848105 12550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:29:09.866036 12561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:09.867123 12560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:09.872980 12563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:09.873221 12550 server_base.cc:1034] running on GCE node
I20260926 19:29:09.874238 12550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:29:09.878023 12550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:29:09.879952 12550 hybrid_clock.cc:648] HybridClock initialized: now 1790450949879842 us; error 100 us; skew 500 ppm
I20260926 19:29:09.880672 12550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:29:09 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:29:09.885673 12550 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:29:09.888068 12550 webserver.cc:460] Webserver started at http://127.1.39.1:41907/ using document root <none> and password file <none>
I20260926 19:29:09.889174 12550 fs_manager.cc:359] Metadata directory not provided
I20260926 19:29:09.889424 12550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:29:09.889871 12550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:29:09 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:09.915302 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "7cbbb79d6a9d462981c433d36b865e50"
format_stamp: "Formatted at 2026-09-26 19:29:09 on dist-test-slave-9wb0"
server_key: "54975f423ab6f895c3ef107d2c684cc5"
server_key_iv: "96271c626a7c7a60366ede8b4bb02ed3"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:09.917788 12550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "7cbbb79d6a9d462981c433d36b865e50"
format_stamp: "Formatted at 2026-09-26 19:29:09 on dist-test-slave-9wb0"
server_key: "54975f423ab6f895c3ef107d2c684cc5"
server_key_iv: "96271c626a7c7a60366ede8b4bb02ed3"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:09.934418 12550 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.003s
I20260926 19:29:09.962564 12570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:09.964771 12550 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260926 19:29:09.965054 12550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "7cbbb79d6a9d462981c433d36b865e50"
format_stamp: "Formatted at 2026-09-26 19:29:09 on dist-test-slave-9wb0"
server_key: "54975f423ab6f895c3ef107d2c684cc5"
server_key_iv: "96271c626a7c7a60366ede8b4bb02ed3"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:09.965829 12550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:29:10.019168 12550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:29:10.026942 12550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:29:10.027334 12550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:29:10.028496 12550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:29:10.030036 12550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:29:10.030143 12550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:10.030311 12550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:29:10.030382 12550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:10.119603 12550 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:34005
I20260926 19:29:10.119622 12682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:34005 every 8 connection(s)
I20260926 19:29:10.121181 12550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:29:10.126861 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 12550
I20260926 19:29:10.127074 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:29:10.128000 1180 external_mini_cluster.cc:1428] Setting key 7ebd7568109cd2bfe9c53a57064266ef
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Sep 26 19:29:10 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:29:10.199127 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:10.126447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42593 (local address 127.1.39.1:45637)
0926 19:29:10.155062 (+ 28615us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:10.155141 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:10.159912 (+ 4771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:10.170881 (+ 10969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:10.170897 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:10.171651 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:29:10.172543 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:10.172574 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:10.175419 (+ 2845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:10.175428 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:10.177948 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:10.177968 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:10.181127 (+ 3159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:10.181134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:10.181464 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:10.182249 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:29:10.182314 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:29:10.189044 (+ 6730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:29:10.193850 (+ 4806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:10.193862 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:10.193884 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:10.194572 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:10.195066 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:10.195070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:10.195074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:10.195334 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:10.196073 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:29:10.196082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:29:10.196547 (+ 465us) client_negotiation.cc:769] Sending connection context
0926 19:29:10.196872 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 19:29:10.197648 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25255,"thread_start_us":26676,"threads_started":1}
I20260926 19:29:10.200343 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:10.127549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:45637 (local address 127.1.39.62:42593)
0926 19:29:10.154612 (+ 27063us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:10.154622 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:10.158174 (+ 3552us) server_negotiation.cc:407] Connection header received
0926 19:29:10.170292 (+ 12118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:10.170299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:10.170433 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:10.170591 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:10.174055 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:10.175233 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:10.178131 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:10.178775 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:10.178915 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:10.189607 (+ 10692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:10.189619 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:10.189625 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:10.189694 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:10.193614 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:10.194751 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:10.194757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:10.194761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:10.194924 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:10.195484 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:10.195489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:10.195493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:10.195851 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:10.195994 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:10.199717 (+ 3723us) server_negotiation.cc:299] Negotiation successful
0926 19:29:10.199973 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26834,"thread_start_us":30633,"threads_started":1}
I20260926 19:29:10.202536 12683 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42593
I20260926 19:29:10.203285 12683 heartbeater.cc:464] Registering TS with master...
I20260926 19:29:10.204192 12683 heartbeater.cc:511] Master 127.1.39.62:42593 requested a full tablet report, sending...
I20260926 19:29:10.207038 12470 ts_manager.cc:194] Registered new tserver with Master: 7cbbb79d6a9d462981c433d36b865e50 (127.1.39.1:34005)
I20260926 19:29:10.209167 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:45637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:29:10.220732 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:49207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:29:10 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
Sep 26 19:29:10 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450949, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
W20260926 19:29:10.709468 12689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:29:10.709987 12689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:29:10.710211 12689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:29:10.721922 12689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:29:10.722157 12689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:29:10.722401 12689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:29:10.738636 12689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.39.20:49207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:29:10.740581 12689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:29:10.742321 12689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:29:10.773630 12697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:10.776926 12696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:10.779731 12700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:10.780248 12689 server_base.cc:1034] running on GCE node
I20260926 19:29:10.781236 12689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:29:10.785970 12689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:29:10.787339 12689 hybrid_clock.cc:648] HybridClock initialized: now 1790450950787230 us; error 105 us; skew 500 ppm
I20260926 19:29:10.787786 12689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:29:10 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450950, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:29:10.792303 12689 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:29:10.813200 12689 webserver.cc:460] Webserver started at http://127.1.39.2:43625/ using document root <none> and password file <none>
I20260926 19:29:10.814447 12689 fs_manager.cc:359] Metadata directory not provided
I20260926 19:29:10.814687 12689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:29:10.815160 12689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:29:10 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450950, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:10.851234 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "9e955277c1714ec4bd42f58f8ac6c87c"
format_stamp: "Formatted at 2026-09-26 19:29:10 on dist-test-slave-9wb0"
server_key: "bc5e2fe32dfe74b7dbe60e3b1d532436"
server_key_iv: "5b6e1dc885b609ddf993532f63f2a818"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:10.852384 12689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "9e955277c1714ec4bd42f58f8ac6c87c"
format_stamp: "Formatted at 2026-09-26 19:29:10 on dist-test-slave-9wb0"
server_key: "bc5e2fe32dfe74b7dbe60e3b1d532436"
server_key_iv: "5b6e1dc885b609ddf993532f63f2a818"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:10.878944 12689 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.005s sys 0.005s
I20260926 19:29:10.906183 12713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:10.908255 12689 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.004s
I20260926 19:29:10.908461 12689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "9e955277c1714ec4bd42f58f8ac6c87c"
format_stamp: "Formatted at 2026-09-26 19:29:10 on dist-test-slave-9wb0"
server_key: "bc5e2fe32dfe74b7dbe60e3b1d532436"
server_key_iv: "5b6e1dc885b609ddf993532f63f2a818"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:10.908728 12689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:29:10.963351 12689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:29:10.968425 12689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:29:10.968816 12689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:29:10.970224 12689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:29:10.972157 12689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:29:10.972287 12689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:10.972497 12689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:29:10.972565 12689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:11.021562 12689 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:39097
I20260926 19:29:11.021588 12825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:39097 every 8 connection(s)
I20260926 19:29:11.023093 12689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:29:11.025101 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 12689
I20260926 19:29:11.025308 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:29:11.026015 1180 external_mini_cluster.cc:1428] Setting key 967405c907d45e9df1cc241137790e1c
Sep 26 19:29:11 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450950, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:29:11.064767 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.026751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:34779 (local address 127.1.39.62:42593)
0926 19:29:11.042631 (+ 15880us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.042639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.043873 (+ 1234us) server_negotiation.cc:407] Connection header received
0926 19:29:11.046903 (+ 3030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.046910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.047030 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.047154 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:11.048935 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.049911 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.051852 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.052416 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.052595 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.057572 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:11.057578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:11.057582 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:11.057625 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:11.061109 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.062000 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.062010 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.062016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.062183 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.062707 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.062713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.062715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.063030 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:11.063222 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.064022 (+ 800us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.064289 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15716,"thread_start_us":15889,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:29:11.066346 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.027435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42593 (local address 127.1.39.2:34779)
0926 19:29:11.043631 (+ 16196us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.043702 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.046447 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.047486 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.047497 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.048019 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:29:11.048696 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.048717 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.050130 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.050137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.051628 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.051672 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.052713 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.052722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.053114 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.054374 (+ 1260us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:29:11.054418 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:29:11.057239 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:29:11.061318 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:11.061330 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:11.061351 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:11.061836 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:11.062317 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:11.062324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:11.062329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:11.062558 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:11.063178 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:29:11.063187 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:29:11.063585 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.063897 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.064494 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15831,"thread_start_us":15586,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
I20260926 19:29:11.068126 12826 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42593
I20260926 19:29:11.068473 12826 heartbeater.cc:464] Registering TS with master...
I20260926 19:29:11.069380 12826 heartbeater.cc:511] Master 127.1.39.62:42593 requested a full tablet report, sending...
I20260926 19:29:11.071247 12470 ts_manager.cc:194] Registered new tserver with Master: 9e955277c1714ec4bd42f58f8ac6c87c (127.1.39.2:39097)
I20260926 19:29:11.072685 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:34779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:29:11.104586 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:49207/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:29:11.213549 12683 heartbeater.cc:503] Master 127.1.39.62:42593 was elected leader, sending a full tablet report...
W20260926 19:29:11.487119 12832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:29:11.487432 12832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:29:11.487605 12832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:29:11.494900 12832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:29:11.495055 12832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:29:11.495237 12832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:29:11.505780 12832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.39.20:49207/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:29:11.507110 12832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:29:11.508417 12832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:29:11.523000 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:11.522941 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:11.522897 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:11.523878 12832 server_base.cc:1034] running on GCE node
I20260926 19:29:11.524525 12832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:29:11.525575 12832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:29:11.526786 12832 hybrid_clock.cc:648] HybridClock initialized: now 1790450951526739 us; error 45 us; skew 500 ppm
I20260926 19:29:11.527138 12832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:29:11 dist-test-slave-9wb0 krb5kdc[11994](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450951, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:29:11.530256 12832 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:29:11.531755 12832 webserver.cc:460] Webserver started at http://127.1.39.3:37425/ using document root <none> and password file <none>
I20260926 19:29:11.532459 12832 fs_manager.cc:359] Metadata directory not provided
I20260926 19:29:11.532604 12832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:29:11.532917 12832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:29:11 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450951, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:11.549003 12832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "95e003183b734d11abc6b6076e8bf804"
format_stamp: "Formatted at 2026-09-26 19:29:11 on dist-test-slave-9wb0"
server_key: "a90d951db99553e2c387eba275286e30"
server_key_iv: "bacb84ff4792c33b2e5cd28cdca7d772"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:11.549904 12832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "95e003183b734d11abc6b6076e8bf804"
format_stamp: "Formatted at 2026-09-26 19:29:11 on dist-test-slave-9wb0"
server_key: "a90d951db99553e2c387eba275286e30"
server_key_iv: "bacb84ff4792c33b2e5cd28cdca7d772"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:11.555512 12832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:29:11.572549 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:11.574200 12832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:29:11.574383 12832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "95e003183b734d11abc6b6076e8bf804"
format_stamp: "Formatted at 2026-09-26 19:29:11 on dist-test-slave-9wb0"
server_key: "a90d951db99553e2c387eba275286e30"
server_key_iv: "bacb84ff4792c33b2e5cd28cdca7d772"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:11.574637 12832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:29:11.615218 12832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:29:11.619083 12832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:29:11.619379 12832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:29:11.620401 12832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:29:11.621928 12832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:29:11.622033 12832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:11.622177 12832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:29:11.622247 12832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:11.661222 12832 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38613
I20260926 19:29:11.661259 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38613 every 8 connection(s)
I20260926 19:29:11.662503 12832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:29:11.664414 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 12832
I20260926 19:29:11.664592 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:29:11.665110 1180 external_mini_cluster.cc:1428] Setting key 8327bf3793bf79c8e9adc1885f02441a
Sep 26 19:29:11 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450951, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:29:11.683041 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.665664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:42561 (local address 127.1.39.62:42593)
0926 19:29:11.666203 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.666208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.667339 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 19:29:11.669091 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.669095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.669168 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.669266 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:11.670629 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.671341 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.673018 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.673603 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.673701 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.676818 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:11.676823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:11.676826 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:11.676857 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:11.679948 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.680778 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.680785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.680790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.680924 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.681391 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.681397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.681401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.681671 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:11.681818 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.682456 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.682664 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":297,"threads_started":1}
I20260926 19:29:11.683916 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.666313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42593 (local address 127.1.39.3:42561)
0926 19:29:11.667192 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.667235 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.668854 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.669512 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.669522 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.669921 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:29:11.670482 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.670497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.671540 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.671544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.672895 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.672906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.673814 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.673819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.674007 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.674603 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:29:11.674628 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:29:11.676616 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:29:11.680091 (+ 3475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:11.680100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:11.680115 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:11.680659 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:11.681042 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:29:11.681047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:29:11.681052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:29:11.681247 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:29:11.681804 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:29:11.681812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:29:11.682110 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.682366 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.682842 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":416,"threads_started":1}
I20260926 19:29:11.685262 12960 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42593
I20260926 19:29:11.685595 12960 heartbeater.cc:464] Registering TS with master...
I20260926 19:29:11.686650 12960 heartbeater.cc:511] Master 127.1.39.62:42593 requested a full tablet report, sending...
I20260926 19:29:11.688258 12470 ts_manager.cc:194] Registered new tserver with Master: 95e003183b734d11abc6b6076e8bf804 (127.1.39.3:38613)
I20260926 19:29:11.689088 12470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:42561
I20260926 19:29:11.692800 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:29:11.709685 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.697333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.1.39.62:42593)
0926 19:29:11.697508 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.697514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.698093 (+ 579us) server_negotiation.cc:407] Connection header received
0926 19:29:11.698247 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.698250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.698322 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.698404 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:11.699798 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.700496 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.701975 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.702394 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.702486 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.704781 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:11.704786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:11.704789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:11.704823 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:11.707508 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.708127 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.708131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.708133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.708240 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:11.708575 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:11.708578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:11.708580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:11.708804 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:11.708976 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.709249 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.709462 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:29:11.714128 12470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:29:11.721407 12470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:29:11.755752 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.735626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.1.39.1:34005)
0926 19:29:11.747035 (+ 11409us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.747043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.747084 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:29:11.747184 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.747192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.747470 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.747683 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:29:11.748462 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.749976 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.753188 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.754618 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.754748 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.754975 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.755241 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.755407 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11201,"thread_start_us":11004,"threads_started":1}
I20260926 19:29:11.755839 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.735394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34005 (local address 127.0.0.1:42942)
0926 19:29:11.745943 (+ 10549us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.746007 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.746244 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.747693 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.747700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.747736 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:29:11.748271 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.748286 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.750196 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.750205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.753044 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.753055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.754799 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.754805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.754991 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.755041 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.755179 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.755377 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10148,"mutex_wait_us":25,"thread_start_us":11597,"threads_started":1}
I20260926 19:29:11.756510 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.735809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:39097 (local address 127.0.0.1:59728)
0926 19:29:11.747244 (+ 11435us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.747261 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.747381 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.748368 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.748375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.748403 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:29:11.748799 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.748811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.750723 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.750728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.753591 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.753606 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.755643 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.755649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.755862 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.755909 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.756001 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.756107 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11318,"mutex_wait_us":111,"thread_start_us":9547,"threads_started":1}
I20260926 19:29:11.757030 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.748013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38613 (local address 127.0.0.1:55638)
0926 19:29:11.749173 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.749189 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.749321 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.749844 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.749848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.749871 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:29:11.750297 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.750310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.752551 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.752556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.754464 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.754474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.756368 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.756376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.756537 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.756582 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.756677 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.756789 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":57,"thread_start_us":923,"threads_started":1}
I20260926 19:29:11.757159 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.736516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.1.39.2:39097)
0926 19:29:11.745171 (+ 8655us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.745186 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.747518 (+ 2332us) server_negotiation.cc:407] Connection header received
0926 19:29:11.747660 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.747668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.747876 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.748069 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:29:11.748970 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.750586 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.753751 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.755116 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.755255 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.755420 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.756713 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.756860 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8420,"thread_start_us":8290,"threads_started":1}
I20260926 19:29:11.757882 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.747771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.1.39.3:38613)
0926 19:29:11.748864 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.748872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.749320 (+ 448us) server_negotiation.cc:407] Connection header received
0926 19:29:11.749454 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.749461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.749658 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.749835 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:29:11.750488 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.752381 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.754603 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.756082 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.756205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.756424 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.757417 (+ 993us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.757566 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":784,"threads_started":1}
I20260926 19:29:11.760257 12618 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (DEFAULT_TABLE table=test-table [id=67b5f89df29549e8819bd7a2b10e6cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:29:11.760576 12761 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (DEFAULT_TABLE table=test-table [id=67b5f89df29549e8819bd7a2b10e6cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:29:11.761926 12618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:29:11.761943 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:29:11.763752 12895 tablet_service.cc:1467] Processing CreateTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (DEFAULT_TABLE table=test-table [id=67b5f89df29549e8819bd7a2b10e6cab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:29:11.765074 12895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:29:11.776469 12979 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Bootstrap starting.
I20260926 19:29:11.779897 12979 tablet_bootstrap.cc:654] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Neither blocks nor log segments found. Creating new log.
I20260926 19:29:11.780885 12979 log.cc:824] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Log is configured to *not* fsync() on all Append() calls
I20260926 19:29:11.782279 12980 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Bootstrap starting.
I20260926 19:29:11.783712 12981 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Bootstrap starting.
I20260926 19:29:11.788345 12981 tablet_bootstrap.cc:654] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Neither blocks nor log segments found. Creating new log.
I20260926 19:29:11.789700 12981 log.cc:824] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:29:11.791368 12979 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: No bootstrap required, opened a new log
I20260926 19:29:11.791662 12979 ts_tablet_manager.cc:1397] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260926 19:29:11.791936 12980 tablet_bootstrap.cc:654] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Neither blocks nor log segments found. Creating new log.
I20260926 19:29:11.792824 12981 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: No bootstrap required, opened a new log
I20260926 19:29:11.792963 12980 log.cc:824] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Log is configured to *not* fsync() on all Append() calls
I20260926 19:29:11.793135 12981 ts_tablet_manager.cc:1397] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260926 19:29:11.796201 12980 tablet_bootstrap.cc:492] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: No bootstrap required, opened a new log
I20260926 19:29:11.796458 12980 ts_tablet_manager.cc:1397] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260926 19:29:11.803180 12979 raft_consensus.cc:348] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.803659 12979 raft_consensus.cc:374] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:29:11.803821 12979 raft_consensus.cc:729] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95e003183b734d11abc6b6076e8bf804, State: Initialized, Role: FOLLOWER
I20260926 19:29:11.804185 12981 raft_consensus.cc:348] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.804595 12981 raft_consensus.cc:374] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:29:11.804488 12979 consensus_queue.cc:260] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.804736 12981 raft_consensus.cc:729] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e955277c1714ec4bd42f58f8ac6c87c, State: Initialized, Role: FOLLOWER
I20260926 19:29:11.804636 12980 raft_consensus.cc:348] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.805122 12980 raft_consensus.cc:374] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:29:11.805248 12980 raft_consensus.cc:729] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cbbb79d6a9d462981c433d36b865e50, State: Initialized, Role: FOLLOWER
I20260926 19:29:11.805382 12981 consensus_queue.cc:260] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.806011 12960 heartbeater.cc:503] Master 127.1.39.62:42593 was elected leader, sending a full tablet report...
I20260926 19:29:11.806933 12979 ts_tablet_manager.cc:1428] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260926 19:29:11.807171 12981 ts_tablet_manager.cc:1428] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260926 19:29:11.807389 12826 heartbeater.cc:503] Master 127.1.39.62:42593 was elected leader, sending a full tablet report...
I20260926 19:29:11.807577 12980 consensus_queue.cc:260] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.809371 12980 ts_tablet_manager.cc:1428] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260926 19:29:11.914778 12961 tablet.cc:2369] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:29:11.921909 12684 tablet.cc:2369] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:29:11.943602 12987 raft_consensus.cc:482] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:29:11.943948 12987 raft_consensus.cc:504] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.945919 12987 leader_election.cc:288] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e955277c1714ec4bd42f58f8ac6c87c (127.1.39.2:39097), 95e003183b734d11abc6b6076e8bf804 (127.1.39.3:38613)
I20260926 19:29:11.953384 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.946398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:59485 (local address 127.1.39.3:38613)
0926 19:29:11.946615 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.946625 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.947463 (+ 838us) server_negotiation.cc:407] Connection header received
0926 19:29:11.947599 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.947605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.947709 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.947805 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:29:11.948452 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.949268 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.951136 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.952064 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.952165 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.952242 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.952906 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.953047 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:29:11.953521 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.946349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38613 (local address 127.1.39.1:59485)
0926 19:29:11.947338 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.947361 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.947482 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.947835 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.947840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.947863 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:29:11.948301 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.948311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.949466 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.949471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.950987 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.950998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.952542 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.952549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.952719 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.952754 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.952857 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.952991 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":20,"spinlock_wait_cycles":2816,"thread_start_us":580,"threads_started":1}
I20260926 19:29:11.953926 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.946366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:40559 (local address 127.1.39.2:39097)
0926 19:29:11.946570 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:11.946577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:11.947613 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 19:29:11.947757 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:11.947761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:11.947868 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:11.947967 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:29:11.948628 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.949351 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.951384 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.952737 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.952873 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.952957 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:11.953518 (+ 561us) server_negotiation.cc:299] Negotiation successful
0926 19:29:11.953676 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:29:11.954025 12989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:11.946426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:39097 (local address 127.1.39.1:40559)
0926 19:29:11.947469 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0926 19:29:11.947489 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:29:11.947631 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:29:11.948010 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:29:11.948014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:29:11.948053 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:29:11.948452 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.948464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.949513 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.949517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.951230 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:29:11.951245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:29:11.952995 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:29:11.953011 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:29:11.953252 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:11.953323 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 19:29:11.953426 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:29:11.953582 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"mutex_wait_us":4,"spinlock_wait_cycles":1408,"thread_start_us":690,"threads_started":1}
I20260926 19:29:11.954792 12915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0cfb7ca2a4ce69a4a358c9ec6b13d" candidate_uuid: "7cbbb79d6a9d462981c433d36b865e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e003183b734d11abc6b6076e8bf804" is_pre_election: true
I20260926 19:29:11.955082 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0cfb7ca2a4ce69a4a358c9ec6b13d" candidate_uuid: "7cbbb79d6a9d462981c433d36b865e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" is_pre_election: true
I20260926 19:29:11.955247 12915 raft_consensus.cc:2454] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cbbb79d6a9d462981c433d36b865e50 in term 0.
I20260926 19:29:11.955569 12781 raft_consensus.cc:2454] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cbbb79d6a9d462981c433d36b865e50 in term 0.
I20260926 19:29:11.956009 12571 leader_election.cc:302] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cbbb79d6a9d462981c433d36b865e50, 95e003183b734d11abc6b6076e8bf804; no voters:
I20260926 19:29:11.956446 12987 raft_consensus.cc:2781] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:29:11.956590 12987 raft_consensus.cc:482] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:29:11.956663 12987 raft_consensus.cc:3037] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:29:11.958848 12987 raft_consensus.cc:504] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.959441 12987 leader_election.cc:288] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [CANDIDATE]: Term 1 election: Requested vote from peers 9e955277c1714ec4bd42f58f8ac6c87c (127.1.39.2:39097), 95e003183b734d11abc6b6076e8bf804 (127.1.39.3:38613)
I20260926 19:29:11.959853 12781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0cfb7ca2a4ce69a4a358c9ec6b13d" candidate_uuid: "7cbbb79d6a9d462981c433d36b865e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e955277c1714ec4bd42f58f8ac6c87c"
I20260926 19:29:11.960008 12915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91d0cfb7ca2a4ce69a4a358c9ec6b13d" candidate_uuid: "7cbbb79d6a9d462981c433d36b865e50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95e003183b734d11abc6b6076e8bf804"
I20260926 19:29:11.960069 12781 raft_consensus.cc:3037] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:29:11.960214 12915 raft_consensus.cc:3037] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:29:11.963413 12915 raft_consensus.cc:2454] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cbbb79d6a9d462981c433d36b865e50 in term 1.
I20260926 19:29:11.963413 12781 raft_consensus.cc:2454] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cbbb79d6a9d462981c433d36b865e50 in term 1.
I20260926 19:29:11.963805 12574 leader_election.cc:302] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cbbb79d6a9d462981c433d36b865e50, 9e955277c1714ec4bd42f58f8ac6c87c; no voters:
I20260926 19:29:11.964120 12987 raft_consensus.cc:2781] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:29:11.965010 12987 raft_consensus.cc:686] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 1 LEADER]: Becoming Leader. State: Replica: 7cbbb79d6a9d462981c433d36b865e50, State: Running, Role: LEADER
I20260926 19:29:11.965524 12987 consensus_queue.cc:237] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } }
I20260926 19:29:11.969918 12469 catalog_manager.cc:5495] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cbbb79d6a9d462981c433d36b865e50 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "7cbbb79d6a9d462981c433d36b865e50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cbbb79d6a9d462981c433d36b865e50" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 } health_report { overall_health: UNKNOWN } } }
W20260926 19:29:12.025573 12827 tablet.cc:2369] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:29:12.041391 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:12.035568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.1.39.1:34005)
0926 19:29:12.035868 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:12.035873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:12.035890 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:29:12.035977 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:12.035981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:12.036054 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:12.036184 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:29:12.036513 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.037229 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.038813 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.039586 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.039684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:12.040187 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:29:12.040717 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:29:12.040870 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:12.040957 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 19:29:12.041021 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 19:29:12.049655 12915 raft_consensus.cc:1261] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 1 FOLLOWER]: Refusing update from remote peer 7cbbb79d6a9d462981c433d36b865e50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:29:12.049659 12781 raft_consensus.cc:1261] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 1 FOLLOWER]: Refusing update from remote peer 7cbbb79d6a9d462981c433d36b865e50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:29:12.050694 12990 consensus_queue.cc:1035] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e955277c1714ec4bd42f58f8ac6c87c" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:29:12.051033 12987 consensus_queue.cc:1035] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95e003183b734d11abc6b6076e8bf804" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:29:12.063710 12993 mvcc.cc:205] Tried to move back new op lower bound from 7333687099584995328 to 7333687099254644736. Current Snapshot: MvccSnapshot[applied={T|T < 7333687099584995328}]
I20260926 19:29:12.063719 12994 mvcc.cc:205] Tried to move back new op lower bound from 7333687099584995328 to 7333687099254644736. Current Snapshot: MvccSnapshot[applied={T|T < 7333687099584995328}]
I20260926 19:29:12.076166 12469 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:29:12.076449 12469 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:29:12.080191 12469 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffc14654987c4ca3b93d1f06586a1522: Sending DeleteTablet for 3 replicas of tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d
I20260926 19:29:12.081473 12618 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:29:12 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:42942
I20260926 19:29:12.081480 12761 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:29:12 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59728
I20260926 19:29:12.081786 12895 tablet_service.cc:1514] Processing DeleteTablet for tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:29:12 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:55638
I20260926 19:29:12.087067 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 12550
I20260926 19:29:12.095618 13002 tablet_replica.cc:330] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: stopping tablet replica
I20260926 19:29:12.096225 13002 raft_consensus.cc:2229] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:29:12.097072 13002 raft_consensus.cc:2258] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:29:12.101912 13002 ts_tablet_manager.cc:1905] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:29:12.111881 13004 tablet_replica.cc:330] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: stopping tablet replica
I20260926 19:29:12.112011 13002 ts_tablet_manager.cc:1918] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:29:12.112263 13002 log.cc:1196] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/91d0cfb7ca2a4ce69a4a358c9ec6b13d
I20260926 19:29:12.112941 13002 ts_tablet_manager.cc:1939] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 7cbbb79d6a9d462981c433d36b865e50: Deleting consensus metadata
I20260926 19:29:12.114461 13004 raft_consensus.cc:2229] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:29:12.114974 13004 raft_consensus.cc:2258] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:29:12.117660 12454 catalog_manager.cc:4841] TS 7cbbb79d6a9d462981c433d36b865e50 (127.1.39.1:34005): tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (table test-table [id=67b5f89df29549e8819bd7a2b10e6cab]) successfully deleted
Sep 26 19:29:12 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:29:12.131814 12974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:12.088788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.1.39.1:34005)
0926 19:29:12.088983 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:12.089006 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:12.111038 (+ 22032us) server_negotiation.cc:407] Connection header received
0926 19:29:12.111285 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:12.111290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:12.111368 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:12.111478 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:12.113854 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.115106 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.118255 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.118938 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.119046 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:12.124286 (+ 5240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:12.124299 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:12.124319 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:12.124365 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:12.128447 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.129336 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.129346 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.129350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.129513 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.129919 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.129924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.129928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.130402 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:12.130677 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:12.131205 (+ 528us) server_negotiation.cc:299] Negotiation successful
0926 19:29:12.131456 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:29:12.133440 12678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:29:12.136664 13004 ts_tablet_manager.cc:1905] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:29:12.141558 13003 tablet_replica.cc:330] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: stopping tablet replica
I20260926 19:29:12.142103 13003 raft_consensus.cc:2229] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:29:12.142625 13003 raft_consensus.cc:2258] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:29:12.144263 13004 ts_tablet_manager.cc:1918] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:29:12.144438 13004 log.cc:1196] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/91d0cfb7ca2a4ce69a4a358c9ec6b13d
I20260926 19:29:12.144878 13004 ts_tablet_manager.cc:1939] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 95e003183b734d11abc6b6076e8bf804: Deleting consensus metadata
I20260926 19:29:12.144862 13003 ts_tablet_manager.cc:1905] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:29:12.146425 12454 catalog_manager.cc:4841] TS 95e003183b734d11abc6b6076e8bf804 (127.1.39.3:38613): tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (table test-table [id=67b5f89df29549e8819bd7a2b10e6cab]) successfully deleted
I20260926 19:29:12.152269 13003 ts_tablet_manager.cc:1918] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:29:12.152501 13003 log.cc:1196] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/91d0cfb7ca2a4ce69a4a358c9ec6b13d
I20260926 19:29:12.153064 13003 ts_tablet_manager.cc:1939] T 91d0cfb7ca2a4ce69a4a358c9ec6b13d P 9e955277c1714ec4bd42f58f8ac6c87c: Deleting consensus metadata
I20260926 19:29:12.155464 12457 catalog_manager.cc:4841] TS 9e955277c1714ec4bd42f58f8ac6c87c (127.1.39.2:39097): tablet 91d0cfb7ca2a4ce69a4a358c9ec6b13d (table test-table [id=67b5f89df29549e8819bd7a2b10e6cab]) successfully deleted
I20260926 19:29:12.241318 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 12550
I20260926 19:29:12.259805 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 12689
Sep 26 19:29:12 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:29:12.274679 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:12.260533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.1.39.2:39097)
0926 19:29:12.260752 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:12.260758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:12.260789 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:29:12.260964 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:12.260970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:12.261067 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:12.261175 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:12.262407 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.263153 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.264459 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.264858 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.265001 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:12.269144 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:12.269167 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:12.269184 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:12.269219 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:12.271925 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.272572 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.272588 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.272592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.272714 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.273115 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.273122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.273125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.273548 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:12.273830 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:12.274127 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:29:12.274336 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 19:29:12.275339 12821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:29:12.366789 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 12689
I20260926 19:29:12.389662 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 12832
Sep 26 19:29:12 dist-test-slave-9wb0 krb5kdc[11994](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:29:12.405860 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:12.390457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.1.39.3:38613)
0926 19:29:12.390633 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:12.390642 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:12.390725 (+ 83us) server_negotiation.cc:407] Connection header received
0926 19:29:12.390940 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:12.390944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:12.391010 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:12.391112 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:12.392433 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.393125 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.394511 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:12.394930 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:12.395065 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:12.399381 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:12.399399 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:12.399422 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:12.399472 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:12.403296 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.403941 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.403950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.403954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.404130 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:12.404480 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:12.404485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:12.404489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:12.404872 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:12.405117 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:12.405404 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 19:29:12.405572 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:29:12.406526 12955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:29:12.503027 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 12832
I20260926 19:29:12.519374 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 12438
I20260926 19:29:12.520581 12500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:29:12.636425 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 12438
I20260926 19:29:12.658865 1180 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 19:29:13.004088 1180 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 19:29:13.004325 1180 mini_ranger.cc:66] Stopping Ranger...
I20260926 19:29:13.349526 1180 mini_ranger.cc:68] Stopped Ranger
2026-09-26 19:29:13.349 UTC [12024] LOG: received smart shutdown request
2026-09-26 19:29:13.353 UTC [12024] LOG: background worker "logical replication launcher" (PID 12033) exited with exit code 1
2026-09-26 19:29:13.353 UTC [12028] LOG: shutting down
2026-09-26 19:29:13.534 UTC [12024] LOG: database system is shut down
2026-09-26T19:29:13Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50108 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:29:13 dist-test-slave-9wb0 krb5kdc[13018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:29:13 dist-test-slave-9wb0 krb5kdc[13018](info): set up 2 sockets
Sep 26 19:29:13 dist-test-slave-9wb0 krb5kdc[13018](info): commencing operation
krb5kdc: starting...
W20260926 19:29:14.830256 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:29:14 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:29:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangeradmin_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangeradmin_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangerlookup_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangerlookup_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
I20260926 19:29:15.166766 1180 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasksjShy7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/postgres -l logfile start
2026-09-26 19:29:17.937 UTC [13048] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 19:29:17.937 UTC [13048] LOG: listening on IPv4 address "127.1.39.20", port 44701
2026-09-26 19:29:17.941 UTC [13048] LOG: listening on Unix socket "/tmp/.s.PGSQL.44701"
2026-09-26 19:29:17.964 UTC [13051] LOG: database system was shut down at 2026-09-26 19:29:16 UTC
2026-09-26 19:29:17.968 UTC [13048] LOG: database system is ready to accept connections
I20260926 19:29:18.953395 1180 mini_postgres.cc:96] Postgres bound to 44701
2026-09-26 19:29:18.971 UTC [13059] FATAL: database "slave" does not exist
127.1.39.20:44701 - accepting connections
I20260926 19:29:18.972524 1180 mini_ranger.cc:161] Starting Ranger...
I20260926 19:29:19.000079 1180 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 19:29:19.571619 1180 mini_ranger.cc:87] Created ranger Postgres database
I20260926 19:29:19.571713 1180 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin
2026-09-26 19:29:19,861 [I] DB FLAVOR :POSTGRES
2026-09-26 19:29:19,861 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:29:19,861 [I] Checking connection..
2026-09-26 19:29:19,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 19:29:20,104 [I] Checking connection passed.
2026-09-26 19:29:20,104 [I] --------- Verifying version history table ---------
2026-09-26 19:29:20,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:29:20,362 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:29:20,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:29:20,622 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 19:29:20,622 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 19:29:20,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 19:29:20.888 UTC [13123] WARNING: there is no transaction in progress
2026-09-26 19:29:20,896 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 19:29:20,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 19:29:21,136 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 19:29:21,136 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:29:21,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 19:29:21,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 19:29:21,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9wb0.c.gcp-upstream.internal','N') ;"
2026-09-26 19:29:21,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:29:22,126 [I] Table x_portal_user does not exist in database ranger
2026-09-26 19:29:22,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:29:22,355 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 19:29:22,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:29:22,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:29:22,863 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 19:29:22,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 19:29:23.912 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:23.929 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:23.947 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:23.964 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:23.981 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.001 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.066 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.073 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.090 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.102 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.114 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.129 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.140 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.147 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.155 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24.164 UTC [13231] WARNING: there is no transaction in progress
2026-09-26 19:29:24,917 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 19:29:24,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 19:29:25,166 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 19:29:25,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 19:29:25,415 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 19:29:25,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 19:29:25,688 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 19:29:25,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 19:29:25,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 19:29:26,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9wb0.c.gcp-upstream.internal';"
2026-09-26 19:29:26,481 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 19:29:26,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 19:29:26,747 [I] DB_PATCHES have already been applied
I20260926 19:29:26.753165 1180 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasksjShy7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 19:29:26.753288 1180 mini_ranger.cc:193] Using host: 127.1.39.20
I20260926 19:29:26.768972 1180 mini_ranger.cc:227] Ranger admin URL: http://127.1.39.20:43877
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:29:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 19:29:27.921109 1180 mini_ranger.cc:160] Time spent starting Ranger: real 8.949s user 0.002s sys 0.041s
Sep 26, 2026 7:29:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:29:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 7:29:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:29:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:29:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 19:29:50.047207 1180 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.39.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.39.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangerkms_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.39.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/rangerkms_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.39.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.39.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.39.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/HTTP_127.1.39.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 19:29:50 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 19:29:50.242873 1180 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 19:29:50.265813 1180 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 19:29:51.262161 1180 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 19:29:51.262287 1180 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms
I20260926 19:29:51.262332 1180 mini_ranger_kms.cc:227] Using postgres at 127.1.39.20:44701
2026-09-26 19:29:51,620 [I] DB FLAVOR :POSTGRES
2026-09-26 19:29:51,620 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 19:29:51,620 [I] Checking connection
2026-09-26 19:29:51,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 19:29:51,860 [I] connection success
2026-09-26 19:29:51,860 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 19:29:51,860 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 19:29:51,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 19:29:52,126 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 19:29:52,126 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 19:29:52,127 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 19:29:52,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.39.20:44701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 19:29:52,389 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 19:29:52.888141 1180 mini_ranger_kms.cc:310] Created kms service
I20260926 19:29:53.092620 1180 mini_ranger_kms.cc:326] Created kudu user
I20260926 19:29:53.140803 1180 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 19:29:53 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:53.590879 1180 mini_ranger_kms.cc:384] Added ranger policy
I20260926 19:29:53.591323 1180 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms:/tmp/dist-test-tasksjShy7/build/asan/bin/postgresql.jar:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasksjShy7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 19:29:53.591398 1180 mini_ranger_kms.cc:240] Using host: 127.1.39.20
I20260926 19:29:53.611886 1180 mini_ranger_kms.cc:276] Ranger KMS PID: 13412
I20260926 19:29:53.612047 1180 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.39.20:38777
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:29:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 19:29:56.021571 1180 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.779s user 0.057s sys 0.321s
I20260926 19:29:56.021809 1180 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 19:29:56.021973 1180 mini_ranger_kms.cc:401] 127.1.39.20:38777/kms/v1/keys
Sep 26 19:29:56 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790450996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 19:29:56 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
Sep 26 19:29:56 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.39.20@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
Sep 26 19:29:59 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:29:59.329376 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:36285
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:36285
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:29:59.667507 13462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:29:59.667829 13462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:29:59.667973 13462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:29:59.675760 13462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:29:59.675913 13462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:29:59.676013 13462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:29:59.676105 13462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:29:59.676189 13462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:29:59.686970 13462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.39.20:38777/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:36285
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:29:59.688320 13462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:29:59.689898 13462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:29:59.705145 13467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:59.705137 13468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:29:59.706538 13470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:29:59.707031 13462 server_base.cc:1034] running on GCE node
I20260926 19:29:59.707938 13462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:29:59.709510 13462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:29:59.710741 13462 hybrid_clock.cc:648] HybridClock initialized: now 1790450999710695 us; error 42 us; skew 500 ppm
I20260926 19:29:59.711105 13462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:29:59 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:29:59.714447 13462 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:29:59.716073 13462 webserver.cc:460] Webserver started at http://127.1.39.62:38821/ using document root <none> and password file <none>
I20260926 19:29:59.716765 13462 fs_manager.cc:359] Metadata directory not provided
I20260926 19:29:59.716902 13462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:29:59.717217 13462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:29:59 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:29:59.784289 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "ecae024a78d44207b16db05fe6f64135"
format_stamp: "Formatted at 2026-09-26 19:29:59 on dist-test-slave-9wb0"
server_key: "54f9debf1700f054c891ec1205f4e74d"
server_key_iv: "ccc5fdbce7e8403abc7fa85dfbdb123c"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:59.785120 13462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "ecae024a78d44207b16db05fe6f64135"
format_stamp: "Formatted at 2026-09-26 19:29:59 on dist-test-slave-9wb0"
server_key: "54f9debf1700f054c891ec1205f4e74d"
server_key_iv: "ccc5fdbce7e8403abc7fa85dfbdb123c"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:59.790552 13462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260926 19:29:59.811017 13477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:29:59.812593 13462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:29:59.812799 13462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "ecae024a78d44207b16db05fe6f64135"
format_stamp: "Formatted at 2026-09-26 19:29:59 on dist-test-slave-9wb0"
server_key: "54f9debf1700f054c891ec1205f4e74d"
server_key_iv: "ccc5fdbce7e8403abc7fa85dfbdb123c"
server_key_version: "kuduclusterkey@0"
I20260926 19:29:59.813009 13462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:29:59.826388 13462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:29:59.830420 13462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:29:59.830745 13462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:29:59.850981 13462 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:36285
I20260926 19:29:59.851001 13528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:36285 every 8 connection(s)
I20260926 19:29:59.852380 13462 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:29:59.856261 13529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:29:59.857335 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 13462
I20260926 19:29:59.857548 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:29:59.858347 1180 external_mini_cluster.cc:1428] Setting key 7ed3f4953d2ada7ee2bbc6382fdecd67
I20260926 19:29:59.874655 13529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Bootstrap starting.
I20260926 19:29:59.878288 13529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Neither blocks nor log segments found. Creating new log.
I20260926 19:29:59.879594 13529 log.cc:824] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Log is configured to *not* fsync() on all Append() calls
I20260926 19:29:59.883005 13529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: No bootstrap required, opened a new log
Sep 26 19:29:59 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:29:59.891917 13529 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:29:59.892351 13529 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:29:59.892482 13529 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecae024a78d44207b16db05fe6f64135, State: Initialized, Role: FOLLOWER
I20260926 19:29:59.893242 13529 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:29:59.893549 13529 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:29:59.893728 13529 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:29:59.893936 13529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:29:59.896909 13529 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:29:59.897583 13529 leader_election.cc:302] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecae024a78d44207b16db05fe6f64135; no voters:
I20260926 19:29:59.899914 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:29:59.860525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.1.39.62:36285)
0926 19:29:59.875304 (+ 14779us) server_negotiation.cc:206] Beginning negotiation
0926 19:29:59.875321 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:29:59.875376 (+ 55us) server_negotiation.cc:407] Connection header received
0926 19:29:59.877894 (+ 2518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:29:59.877924 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:29:59.878458 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:29:59.878914 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:29:59.880607 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:59.882481 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:59.884382 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:29:59.885524 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:29:59.885731 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:29:59.890583 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:29:59.890593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:29:59.890614 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:29:59.890665 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:29:59.894711 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:59.895563 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:59.895572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:59.895585 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:59.895766 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:29:59.896212 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:29:59.896217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:29:59.896220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:29:59.896997 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:29:59.897339 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 19:29:59.898195 (+ 856us) server_negotiation.cc:299] Negotiation successful
0926 19:29:59.898604 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14495,"thread_start_us":16704,"threads_started":1}
I20260926 19:29:59.901919 13529 leader_election.cc:288] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:29:59.902034 13534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:29:59.907655 13534 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 LEADER]: Becoming Leader. State: Replica: ecae024a78d44207b16db05fe6f64135, State: Running, Role: LEADER
I20260926 19:29:59.908264 13534 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:29:59.909034 13529 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:29:59.913367 13536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecae024a78d44207b16db05fe6f64135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } } }
I20260926 19:29:59.913762 13536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: This master's current role is: LEADER
I20260926 19:29:59.914283 13535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecae024a78d44207b16db05fe6f64135. Latest consensus state: current_term: 1 leader_uuid: "ecae024a78d44207b16db05fe6f64135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } } }
I20260926 19:29:59.914562 13535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: This master's current role is: LEADER
I20260926 19:29:59.914746 13540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:29:59.933446 13540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:29:59.952060 13540 catalog_manager.cc:1347] Generated new cluster ID: 9950c0791b084652bb1a1a1ef81b6804
I20260926 19:29:59.952234 13540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:29:59.974191 13540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:29:59.975131 13529 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 19:29:59.976181 13540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:29:59.988657 13540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Generated new TSK 0
I20260926 19:29:59.990916 13540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:30:00.893539 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:01 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 19:30:01.323251 13574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:01.323659 13574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:01.323863 13574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:01.335516 13574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:01.335671 13574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:01.335911 13574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:01.349505 13574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:01.351032 13574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:01.352630 13574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:01.378229 13584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:01.380141 13585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:01.385226 13587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:01.385639 13574 server_base.cc:1034] running on GCE node
I20260926 19:30:01.386430 13574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:01.389585 13574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:01.390913 13574 hybrid_clock.cc:648] HybridClock initialized: now 1790451001390828 us; error 70 us; skew 500 ppm
I20260926 19:30:01.391319 13574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:01 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:01.395069 13574 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:01.396821 13574 webserver.cc:460] Webserver started at http://127.1.39.1:34209/ using document root <none> and password file <none>
I20260926 19:30:01.397747 13574 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:01.397877 13574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:01.398216 13574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:30:01 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:01.418716 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "b564cfe6f9734b139105287e27e4089c"
format_stamp: "Formatted at 2026-09-26 19:30:01 on dist-test-slave-9wb0"
server_key: "3ff425c4d10bac41256f5e253b87676a"
server_key_iv: "a51b4114f012d3ffbf87cb7858ee1b8c"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:01.419932 13574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "b564cfe6f9734b139105287e27e4089c"
format_stamp: "Formatted at 2026-09-26 19:30:01 on dist-test-slave-9wb0"
server_key: "3ff425c4d10bac41256f5e253b87676a"
server_key_iv: "a51b4114f012d3ffbf87cb7858ee1b8c"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:01.438169 13574 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.014s sys 0.005s
I20260926 19:30:01.469980 13594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:01.475129 13574 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.002s
I20260926 19:30:01.475479 13574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "b564cfe6f9734b139105287e27e4089c"
format_stamp: "Formatted at 2026-09-26 19:30:01 on dist-test-slave-9wb0"
server_key: "3ff425c4d10bac41256f5e253b87676a"
server_key_iv: "a51b4114f012d3ffbf87cb7858ee1b8c"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:01.475862 13574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:01.506403 13574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:01.512516 13574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:01.512898 13574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:01.515802 13574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:01.518770 13574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:01.518884 13574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:01.519083 13574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:01.519160 13574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:01.596680 13574 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:42559
I20260926 19:30:01.596719 13706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:42559 every 8 connection(s)
I20260926 19:30:01.598685 13574 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:01.606494 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 13574
I20260926 19:30:01.606725 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:30:01.607544 1180 external_mini_cluster.cc:1428] Setting key 15de0feefb21866b0f45740f11ad4d40
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Sep 26 19:30:01 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
Sep 26 19:30:01 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:01.674048 13710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:01.602837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.1:54345)
0926 19:30:01.626425 (+ 23588us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:01.626497 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:01.633964 (+ 7467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:01.635406 (+ 1442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:01.635423 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:01.636818 (+ 1395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:01.637910 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:01.637942 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:01.642678 (+ 4736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:01.642685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:01.646187 (+ 3502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:01.646224 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:01.649580 (+ 3356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:01.649590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:01.650023 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:01.650804 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:01.650840 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:01.663040 (+ 12200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:01.668596 (+ 5556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:01.668606 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:01.668628 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:01.669370 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:01.669872 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:01.669877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:01.669881 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:01.670146 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:01.670935 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:01.670944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:01.671390 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 19:30:01.671747 (+ 357us) client_negotiation.cc:240] Negotiation successful
0926 19:30:01.672339 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23201,"thread_start_us":28493,"threads_started":1}
I20260926 19:30:01.675536 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:01.601674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:54345 (local address 127.1.39.62:36285)
0926 19:30:01.629099 (+ 27425us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:01.629106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:01.629134 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:30:01.634578 (+ 5444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:01.634586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:01.634718 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:01.634850 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:01.638323 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:01.640741 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:01.646972 (+ 6231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:01.647678 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:01.647797 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:01.663429 (+ 15632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:01.663440 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:01.663445 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:01.663512 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:01.667468 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:01.669564 (+ 2096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:01.669571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:01.669574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:01.669731 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:01.670311 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:01.670317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:01.670321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:01.670711 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:01.670856 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:01.674863 (+ 4007us) server_negotiation.cc:299] Negotiation successful
0926 19:30:01.675143 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27253,"thread_start_us":27462,"threads_started":1}
I20260926 19:30:01.678141 13707 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:01.678505 13707 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:01.679391 13707 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:01.682622 13494 ts_manager.cc:194] Registered new tserver with Master: b564cfe6f9734b139105287e27e4089c (127.1.39.1:42559)
I20260926 19:30:01.684857 13494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:54345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
W20260926 19:30:01.721863 1180 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.1.39.2: real 0.111s user 0.001s sys 0.018s
I20260926 19:30:01.723222 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451001, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
W20260926 19:30:02.290344 13713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:02.290870 13713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:02.291064 13713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:02.303903 13713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:02.304673 13713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:02.305155 13713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:02.321892 13713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:02.324220 13713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:02.326342 13713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:02.350747 13720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:02.351025 13721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:02.359891 13723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:02.360221 13713 server_base.cc:1034] running on GCE node
I20260926 19:30:02.361498 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:02.369655 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:02.373178 13713 hybrid_clock.cc:648] HybridClock initialized: now 1790451002373008 us; error 167 us; skew 500 ppm
I20260926 19:30:02.373762 13713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:02.378649 13713 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:02.385529 13713 webserver.cc:460] Webserver started at http://127.1.39.2:36163/ using document root <none> and password file <none>
I20260926 19:30:02.386590 13713 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:02.386797 13713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:02.387262 13713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:02.404763 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "615780e398d545d48484f8b2250f6fe7"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "10cbcb1b2fee3787e30689c2e7ea0993"
server_key_iv: "b876fd7c3200a00cd046cdf7b29a90a1"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:02.405736 13713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "615780e398d545d48484f8b2250f6fe7"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "10cbcb1b2fee3787e30689c2e7ea0993"
server_key_iv: "b876fd7c3200a00cd046cdf7b29a90a1"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:02.411876 13713 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 19:30:02.426970 13737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:02.428623 13713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:30:02.428828 13713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "615780e398d545d48484f8b2250f6fe7"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "10cbcb1b2fee3787e30689c2e7ea0993"
server_key_iv: "b876fd7c3200a00cd046cdf7b29a90a1"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:02.429069 13713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:02.447103 13713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:02.451484 13713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:02.451812 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:02.452800 13713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:02.454350 13713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:02.454455 13713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:02.454603 13713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:02.454679 13713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:02.499199 13713 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:36211
I20260926 19:30:02.499217 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:36211 every 8 connection(s)
I20260926 19:30:02.500567 13713 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:02.510401 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 13713
I20260926 19:30:02.510584 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:30:02.511047 1180 external_mini_cluster.cc:1428] Setting key 3ae1e13105c41dadc92ca3e8cdc023b9
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:02.522765 13852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:02.503284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:53613 (local address 127.1.39.62:36285)
0926 19:30:02.503762 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:02.503769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:02.504907 (+ 1138us) server_negotiation.cc:407] Connection header received
0926 19:30:02.507349 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:02.507354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:02.507440 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:02.507551 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:02.509449 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:02.510252 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:02.512217 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:02.512663 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:02.512796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:02.516310 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:02.516319 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:02.516326 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:02.516385 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:02.519490 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:02.520310 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:02.520315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:02.520317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:02.520448 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:02.520929 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:02.520933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:02.520936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:02.521222 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:02.521415 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:02.522156 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 19:30:02.522378 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260926 19:30:02.524115 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:02.503811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.2:53613)
0926 19:30:02.504716 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:02.504766 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:02.507017 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:02.507823 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:02.507835 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:02.508410 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:02.509207 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:02.509230 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:02.510502 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:02.510511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:02.512023 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:02.512036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:02.512902 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:02.512909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:02.513141 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:02.513903 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:02.513926 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:02.516061 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:02.519667 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:02.519679 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:02.519699 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:02.520164 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:02.520572 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:02.520578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:02.520582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:02.520797 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:02.521394 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:02.521403 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:02.521753 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 19:30:02.522042 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 19:30:02.522582 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":358,"threads_started":1}
I20260926 19:30:02.525862 13850 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:02.526234 13850 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:02.527163 13850 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:02.528957 13494 ts_manager.cc:194] Registered new tserver with Master: 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211)
I20260926 19:30:02.530150 13494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:53613
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:30:02.570667 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:02.689579 13707 heartbeater.cc:503] Master 127.1.39.62:36285 was elected leader, sending a full tablet report...
W20260926 19:30:02.913281 13856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:02.913654 13856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:02.913806 13856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:02.921540 13856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:02.921702 13856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:02.921857 13856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:02.932123 13856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:02.933519 13856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:02.934875 13856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:02.948771 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:02.948756 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:02.948750 13864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:02.949702 13856 server_base.cc:1034] running on GCE node
I20260926 19:30:02.950362 13856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:02.951408 13856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:02.952661 13856 hybrid_clock.cc:648] HybridClock initialized: now 1790451002952613 us; error 38 us; skew 500 ppm
I20260926 19:30:02.953006 13856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:02.956302 13856 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:02.958250 13856 webserver.cc:460] Webserver started at http://127.1.39.3:43525/ using document root <none> and password file <none>
I20260926 19:30:02.959008 13856 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:02.959159 13856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:02.959491 13856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 19:30:02 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:02.975983 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "c3518f1b04a64028af0403675a1458f5"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "08d2d10d3075340c657c5635f17fcd31"
server_key_iv: "62ebc44bae81dfb08e658b7b8d7f3edf"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:02.976933 13856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "c3518f1b04a64028af0403675a1458f5"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "08d2d10d3075340c657c5635f17fcd31"
server_key_iv: "62ebc44bae81dfb08e658b7b8d7f3edf"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:02.983244 13856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 19:30:03.000638 13871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:03.002471 13856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 19:30:03.002681 13856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "c3518f1b04a64028af0403675a1458f5"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "08d2d10d3075340c657c5635f17fcd31"
server_key_iv: "62ebc44bae81dfb08e658b7b8d7f3edf"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:03.002924 13856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:03.018682 13856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:03.022929 13856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:03.023239 13856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:03.024224 13856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:03.025660 13856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:03.025736 13856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:03.025899 13856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:03.025965 13856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:03.060568 13856 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38991
I20260926 19:30:03.060573 13983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38991 every 8 connection(s)
I20260926 19:30:03.062074 13856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:03.068604 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 13856
I20260926 19:30:03.068804 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:30:03.069394 1180 external_mini_cluster.cc:1428] Setting key 22f8fb271a5f1e264f567c1fdb55e71b
Sep 26 19:30:03 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451002, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:03.083556 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:03.064966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:39579 (local address 127.1.39.62:36285)
0926 19:30:03.065477 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:03.065485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:03.066388 (+ 903us) server_negotiation.cc:407] Connection header received
0926 19:30:03.068192 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:03.068196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:03.068281 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:03.068392 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:03.070310 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.071343 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.073206 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.073687 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.073783 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:03.077418 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:03.077424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:03.077427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:03.077465 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:03.080083 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.080838 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.080842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.080845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.080972 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.081634 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.081638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.081640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.082086 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:03.082216 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:03.082928 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 19:30:03.083179 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":226,"threads_started":1}
I20260926 19:30:03.084748 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:03.065520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.3:39579)
0926 19:30:03.066225 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:03.066270 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:03.067937 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:03.068668 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:03.068681 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:03.069264 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:03.070079 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:03.070124 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.071518 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.071523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:03.073065 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:03.073084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.073877 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.073882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:03.074082 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:03.074766 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:03.074791 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:03.077112 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:03.080234 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:03.080242 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:03.080257 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:03.080710 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:03.081174 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:03.081180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:03.081185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:03.081502 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:03.082213 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:03.082220 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:03.082559 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 19:30:03.082829 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 19:30:03.083436 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":288,"threads_started":1}
I20260926 19:30:03.086509 13984 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:03.086858 13984 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:03.087627 13984 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:03.089211 13494 ts_manager.cc:194] Registered new tserver with Master: c3518f1b04a64028af0403675a1458f5 (127.1.39.3:38991)
I20260926 19:30:03.090287 13494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:39579
I20260926 19:30:03.097873 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:30:03.099601 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 13574
Sep 26 19:30:03 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:30:03.119215 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:03.100589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58164 (local address 127.1.39.1:42559)
0926 19:30:03.101567 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:03.101575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:03.101605 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:30:03.101675 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:03.101682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:03.101910 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:03.102056 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:03.103587 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.104663 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.106696 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.107427 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.107543 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:03.112718 (+ 5175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:03.112733 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:03.112750 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:03.112791 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:03.116146 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.116979 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.116986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.116989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.117134 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.117558 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.117562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.117564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.117968 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:03.118202 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:03.118576 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 19:30:03.118826 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":599,"threads_started":1}
I20260926 19:30:03.120814 13702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:03.205693 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 13574
I20260926 19:30:03.223448 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 13713
Sep 26 19:30:03 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:30:03.241614 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:03.224387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.1.39.2:36211)
0926 19:30:03.224994 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:03.225002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:03.225031 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:30:03.225095 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:03.225102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:03.225416 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:03.225566 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:03.227022 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.228094 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.229538 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.230083 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.230199 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:03.234706 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:03.234717 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:03.234733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:03.234779 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:03.238162 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.239066 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.239073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.239076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.239246 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.239813 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.239818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.239820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.240218 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:03.240617 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:03.240873 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 19:30:03.241095 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":321,"threads_started":1}
I20260926 19:30:03.242924 13845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:03.327632 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 13713
I20260926 19:30:03.343911 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 13856
Sep 26 19:30:03 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790450999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:30:03.367020 13993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:03.344930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.1.39.3:38991)
0926 19:30:03.345641 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:03.345648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:03.345677 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:30:03.345734 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:03.345740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:03.345965 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:03.346134 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:03.347493 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.348492 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.350066 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:03.350640 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:03.350782 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:03.361212 (+ 10430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:03.361229 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:03.361245 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:03.361358 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:03.364301 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.365103 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.365110 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.365112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.365237 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:03.365695 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:03.365699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:03.365701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:03.366038 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:03.366264 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:03.366516 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 19:30:03.366733 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":412,"threads_started":1}
I20260926 19:30:03.368266 13979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:03.452569 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 13856
I20260926 19:30:03.469075 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 13462
I20260926 19:30:03.470525 13524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:03.577906 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 13462
I20260926 19:30:03.603562 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:36285
--webserver_interface=127.1.39.62
--webserver_port=38821
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:36285
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:03.959573 13996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:03.959849 13996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:03.959936 13996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:03.967090 13996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:03.967195 13996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:03.967238 13996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:03.967286 13996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:03.967332 13996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:03.977833 13996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:36285
--ranger_config_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=38821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:03.979169 13996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:03.980434 13996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:03.992393 14002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:03.992405 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:03.992921 14001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:03.994345 13996 server_base.cc:1034] running on GCE node
I20260926 19:30:03.995152 13996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:03.996538 13996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:03.997762 13996 hybrid_clock.cc:648] HybridClock initialized: now 1790451003997719 us; error 42 us; skew 500 ppm
I20260926 19:30:03.998114 13996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:03 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:04.001142 13996 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:04.002605 13996 webserver.cc:460] Webserver started at http://127.1.39.62:38821/ using document root <none> and password file <none>
I20260926 19:30:04.003211 13996 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:04.003314 13996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:30:04 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451003, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:04.060312 13996 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:30:04.063504 14012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:04.065145 13996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:30:04.065383 13996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "ecae024a78d44207b16db05fe6f64135"
format_stamp: "Formatted at 2026-09-26 19:29:59 on dist-test-slave-9wb0"
server_key: "54f9debf1700f054c891ec1205f4e74d"
server_key_iv: "ccc5fdbce7e8403abc7fa85dfbdb123c"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:04.066432 13996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:04.083070 13996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:04.087350 13996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:04.087687 13996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:04.106971 13996 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:36285
I20260926 19:30:04.106984 14063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:36285 every 8 connection(s)
I20260926 19:30:04.108578 13996 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:04.110911 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 13996
I20260926 19:30:04.111768 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:42559
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:36285
--webserver_port=34209
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:04.116303 14064 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:30:04.119874 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Bootstrap starting.
I20260926 19:30:04.140424 14064 log.cc:824] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:04.149647 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:30:04.150231 14064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Bootstrap complete.
I20260926 19:30:04.160228 14064 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:30:04.161201 14064 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecae024a78d44207b16db05fe6f64135, State: Initialized, Role: FOLLOWER
I20260926 19:30:04.161985 14064 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:30:04.162261 14064 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:04.162405 14064 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:04.162601 14064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:30:04.166842 14064 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:30:04.167431 14064 leader_election.cc:302] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecae024a78d44207b16db05fe6f64135; no voters:
I20260926 19:30:04.168119 14064 leader_election.cc:288] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:30:04.168284 14068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:30:04.169538 14068 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [term 2 LEADER]: Becoming Leader. State: Replica: ecae024a78d44207b16db05fe6f64135, State: Running, Role: LEADER
I20260926 19:30:04.170233 14068 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } }
I20260926 19:30:04.170584 14064 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:04.173204 14070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecae024a78d44207b16db05fe6f64135. Latest consensus state: current_term: 2 leader_uuid: "ecae024a78d44207b16db05fe6f64135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } } }
I20260926 19:30:04.173629 14070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:04.173341 14069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ecae024a78d44207b16db05fe6f64135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecae024a78d44207b16db05fe6f64135" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 36285 } } }
I20260926 19:30:04.173774 14069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:04.184197 14074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:04.191442 14074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:04.193516 14074 catalog_manager.cc:1259] Loaded cluster ID: 9950c0791b084652bb1a1a1ef81b6804
I20260926 19:30:04.193697 14074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:04.199127 14074 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:04.201328 14074 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Loaded TSK: 0
I20260926 19:30:04.203554 14074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:30:04.208043 14064 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260926 19:30:04.560132 14066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:04.560665 14066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:04.560873 14066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:04.573398 14066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:04.573563 14066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:04.573808 14066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:04.590562 14066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:42559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=34209
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:04.592483 14066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:04.594332 14066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:04.634486 14102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:04.637784 14101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:04.649746 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:04.651315 14066 server_base.cc:1034] running on GCE node
I20260926 19:30:04.652179 14066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:04.662261 14066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:04.663698 14066 hybrid_clock.cc:648] HybridClock initialized: now 1790451004663631 us; error 74 us; skew 500 ppm
I20260926 19:30:04.664180 14066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:04 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451004, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:04.668674 14066 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:04.676800 14066 webserver.cc:460] Webserver started at http://127.1.39.1:34209/ using document root <none> and password file <none>
I20260926 19:30:04.677906 14066 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:04.678079 14066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:30:04 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451004, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:04.727608 14066 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.008s sys 0.008s
I20260926 19:30:04.731824 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:04.733780 14066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 19:30:04.734066 14066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "b564cfe6f9734b139105287e27e4089c"
format_stamp: "Formatted at 2026-09-26 19:30:01 on dist-test-slave-9wb0"
server_key: "3ff425c4d10bac41256f5e253b87676a"
server_key_iv: "a51b4114f012d3ffbf87cb7858ee1b8c"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:04.735702 14066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:04.779210 14066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:04.784265 14066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:04.784606 14066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:04.787225 14066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:04.795542 14066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:04.795708 14066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:04.795854 14066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:04.795990 14066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:04.884142 14066 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:42559
I20260926 19:30:04.884224 14225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:42559 every 8 connection(s)
I20260926 19:30:04.886083 14066 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:04.894680 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14066
I20260926 19:30:04.895725 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:36211
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:36285
--webserver_port=36163
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:04 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451004, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:04.931631 14229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:04.889461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.1:59663)
0926 19:30:04.902160 (+ 12699us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:04.902220 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:04.905165 (+ 2945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:04.908342 (+ 3177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:04.908355 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:04.909001 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:04.909791 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:04.909817 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:04.912516 (+ 2699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:04.912525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:04.914844 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:04.914863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:04.916816 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:04.916825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:04.917152 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:04.918049 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:04.918083 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:04.921152 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:04.926410 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:04.926421 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:04.926440 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:04.927138 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:04.927752 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:04.927757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:04.927760 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:04.927994 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:04.929036 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:04.929044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:04.929441 (+ 397us) client_negotiation.cc:769] Sending connection context
0926 19:30:04.929733 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 19:30:04.930308 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12334,"thread_start_us":16091,"threads_started":1}
I20260926 19:30:04.933737 14228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:04.889421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:59663 (local address 127.1.39.62:36285)
0926 19:30:04.901204 (+ 11783us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:04.901230 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:04.905679 (+ 4449us) server_negotiation.cc:407] Connection header received
0926 19:30:04.906188 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:04.906207 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:04.906690 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:04.907097 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:04.910037 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:04.912302 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:04.915038 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:04.916205 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:04.916366 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:04.921558 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:04.921573 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:04.921602 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:04.921666 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:04.926203 (+ 4537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:04.927329 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:04.927340 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:04.927348 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:04.927573 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:04.928134 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:04.928139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:04.928143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:04.928754 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:04.928933 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:04.932111 (+ 3178us) server_negotiation.cc:299] Negotiation successful
0926 19:30:04.932514 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11397,"thread_start_us":11743,"threads_started":1}
I20260926 19:30:04.937602 14226 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:04.937948 14226 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:04.938694 14226 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
W20260926 19:30:04.941905 14226 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:36285 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 19:30:04.942128 14226 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:04.942412 14226 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:04.943696 14226 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:04.943944 14226 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
W20260926 19:30:04.945048 14226 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:30:05.321976 14230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:05.322306 14230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:05.322484 14230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:05.330103 14230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:05.330274 14230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:05.330466 14230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:05.340546 14230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:36211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=36163
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:05.342115 14230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:05.343608 14230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:05.371434 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:05.381441 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:05.383044 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:05.387274 14230 server_base.cc:1034] running on GCE node
I20260926 19:30:05.388258 14230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:05.389595 14230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:05.390833 14230 hybrid_clock.cc:648] HybridClock initialized: now 1790451005390766 us; error 67 us; skew 500 ppm
I20260926 19:30:05.391191 14230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:05 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:05.395661 14230 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:05.412608 14230 webserver.cc:460] Webserver started at http://127.1.39.2:36163/ using document root <none> and password file <none>
I20260926 19:30:05.413498 14230 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:05.413721 14230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:30:05 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:05.439114 14230 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260926 19:30:05.443239 14257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:05.444785 14230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 19:30:05.444983 14230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "615780e398d545d48484f8b2250f6fe7"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "10cbcb1b2fee3787e30689c2e7ea0993"
server_key_iv: "b876fd7c3200a00cd046cdf7b29a90a1"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:05.446117 14230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 19:30:05 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:05.488943 14230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:05.494573 14230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:05.494958 14230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:05.498574 14230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:05.507406 14230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:05.507534 14230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:05.507663 14230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:05.507743 14230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:05.553399 14230 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:36211
I20260926 19:30:05.553480 14369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:36211 every 8 connection(s)
I20260926 19:30:05.555012 14230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:05.564229 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14230
I20260926 19:30:05.565018 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.39.20:38777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:38991
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:36285
--webserver_port=43525
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:05 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:05.584685 14373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:05.559786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:50555 (local address 127.1.39.62:36285)
0926 19:30:05.560402 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:05.560410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:05.563509 (+ 3099us) server_negotiation.cc:407] Connection header received
0926 19:30:05.563589 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:05.563602 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:05.563717 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:05.563834 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:05.566162 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:05.567375 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:05.570094 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:05.570978 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:05.571107 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:05.576270 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:05.576278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:05.576286 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:05.576339 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:05.580327 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:05.581461 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:05.581471 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:05.581477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:05.581632 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:05.582234 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:05.582242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:05.582248 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:05.582653 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:05.582795 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:05.583887 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0926 19:30:05.584181 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":282,"threads_started":1}
I20260926 19:30:05.587675 14374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:05.560412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.2:50555)
0926 19:30:05.561203 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:05.561250 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:05.563133 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:05.564352 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:05.564367 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:05.565009 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:05.565925 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:05.565955 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:05.567582 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:05.567591 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:05.569899 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:05.569917 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:05.571556 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:05.571566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:05.571952 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:05.572804 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:05.572840 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:05.575909 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:05.580542 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:05.580554 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:05.580578 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:05.581286 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:05.581786 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:05.581793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:05.581799 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:05.582095 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:05.582899 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:05.582910 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:05.583414 (+ 504us) client_negotiation.cc:769] Sending connection context
0926 19:30:05.585331 (+ 1917us) client_negotiation.cc:240] Negotiation successful
0926 19:30:05.586015 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":627,"threads_started":1}
I20260926 19:30:05.590402 14371 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:05.590749 14371 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:05.591806 14371 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:05.594247 14029 ts_manager.cc:194] Registered new tserver with Master: 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211)
I20260926 19:30:05.596439 14029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:50555
Sep 26 19:30:05 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:05.945201 14226 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:05.945684 14226 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:05.947053 14029 ts_manager.cc:194] Registered new tserver with Master: b564cfe6f9734b139105287e27e4089c (127.1.39.1:42559)
I20260926 19:30:05.948262 14029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:59663
W20260926 19:30:06.013200 14375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:06.013731 14375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:06.013955 14375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:06.025813 14375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:06.026034 14375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:06.026257 14375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:06.042042 14375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.39.20:38777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:38991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=43525
--tserver_master_addrs=127.1.39.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:06.043735 14375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:06.045490 14375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:06.095173 14380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:06.097033 14381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:06.105345 14384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:06.105850 14375 server_base.cc:1034] running on GCE node
I20260926 19:30:06.106950 14375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:06.113493 14375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:06.114828 14375 hybrid_clock.cc:648] HybridClock initialized: now 1790451006114768 us; error 64 us; skew 500 ppm
I20260926 19:30:06.115271 14375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:06 dist-test-slave-9wb0 krb5kdc[13018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:06.120251 14375 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:06.159525 14375 webserver.cc:460] Webserver started at http://127.1.39.3:43525/ using document root <none> and password file <none>
I20260926 19:30:06.160688 14375 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:06.161024 14375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 19:30:06 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:06.207789 14375 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.003s
I20260926 19:30:06.211853 14391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:06.213862 14375 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 19:30:06.214071 14375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "c3518f1b04a64028af0403675a1458f5"
format_stamp: "Formatted at 2026-09-26 19:30:02 on dist-test-slave-9wb0"
server_key: "08d2d10d3075340c657c5635f17fcd31"
server_key_iv: "62ebc44bae81dfb08e658b7b8d7f3edf"
server_key_version: "kuduclusterkey@0"
I20260926 19:30:06.215462 14375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:06.243036 14375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:06.248423 14375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:06.248735 14375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:06.250149 14375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:06.252146 14375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:06.252270 14375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:06.252393 14375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:06.252476 14375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 19:30:06 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451005, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for HTTP/127.1.39.20@KRBTEST.COM
I20260926 19:30:06.532928 14375 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38991
I20260926 19:30:06.532972 14504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38991 every 8 connection(s)
I20260926 19:30:06.534623 14375 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:06.535944 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14375
Sep 26 19:30:06 dist-test-slave-9wb0 krb5kdc[13018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451006, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:06.564379 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.536931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.1.39.62:36285)
0926 19:30:06.540478 (+ 3547us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.540488 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.541948 (+ 1460us) server_negotiation.cc:407] Connection header received
0926 19:30:06.542072 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.542080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.542190 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.542318 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:06.547414 (+ 5096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.548679 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.550841 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.551576 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.551709 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.555490 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:06.555499 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:06.555508 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:06.555595 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:06.559291 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.560294 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.560304 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.560308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.560473 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.560931 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.560936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.560939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.561642 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:06.561924 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.563756 (+ 1832us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.563989 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3368,"thread_start_us":3249,"threads_started":1}
I20260926 19:30:06.573246 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.538600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:36285 (local address 127.1.39.3:45137)
0926 19:30:06.548459 (+ 9859us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.548517 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.551943 (+ 3426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.553180 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.553193 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.553937 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:06.554830 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.554858 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.556659 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.556667 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.559260 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.559276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.560601 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.560609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.560921 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.561685 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:06.561721 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:06.564259 (+ 2538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:06.569032 (+ 4773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:06.569041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:06.569059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:06.569659 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:06.570077 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:06.570082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:06.570085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:06.570303 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:06.571057 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:06.571069 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:06.571444 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.571744 (+ 300us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.572231 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9518,"thread_start_us":10561,"threads_started":1}
I20260926 19:30:06.574347 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.540384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:45137 (local address 127.1.39.62:36285)
0926 19:30:06.547003 (+ 6619us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.547012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.548803 (+ 1791us) server_negotiation.cc:407] Connection header received
0926 19:30:06.552415 (+ 3612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.552423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.552517 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.552622 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:06.555089 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.556446 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.559426 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.560080 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.560182 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.564548 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:06.564553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:06.564558 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:06.564595 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:06.568896 (+ 4301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.569783 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.569788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.569792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.569959 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.570429 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.570434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.570437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.570871 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:06.571025 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.573848 (+ 2823us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.574088 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6420,"spinlock_wait_cycles":26880,"thread_start_us":6529,"threads_started":1}
I20260926 19:30:06.575930 14505 heartbeater.cc:347] Connected to a master server at 127.1.39.62:36285
I20260926 19:30:06.576269 14505 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:06.576997 14505 heartbeater.cc:511] Master 127.1.39.62:36285 requested a full tablet report, sending...
I20260926 19:30:06.579080 14029 ts_manager.cc:194] Registered new tserver with Master: c3518f1b04a64028af0403675a1458f5 (127.1.39.3:38991)
I20260926 19:30:06.580237 14029 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:45137
I20260926 19:30:06.587793 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:30:06.601560 14371 heartbeater.cc:503] Master 127.1.39.62:36285 was elected leader, sending a full tablet report...
I20260926 19:30:06.615252 14510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.600135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.1.39.62:36285)
0926 19:30:06.600353 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.600360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.600862 (+ 502us) server_negotiation.cc:407] Connection header received
0926 19:30:06.601036 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.601039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.601116 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.601201 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:06.603261 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.603953 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.606031 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.606424 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.606524 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.609863 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:06.609867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:06.609870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:06.609903 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:06.612506 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.613329 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.613337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.613344 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.613491 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:06.613913 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:06.613917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:06.613918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:06.614187 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:06.614458 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.614792 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.614960 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:30:06.621213 14029 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:30:06.631780 14029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:30:06.657348 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.646386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:36211 (local address 127.0.0.1:33416)
0926 19:30:06.647406 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.647459 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.647679 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.649079 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.649091 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.649136 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.649671 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.649686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.651863 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.651871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.654389 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.654403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.656301 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.656308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.656522 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.656604 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.656710 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.656915 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":2924,"threads_started":1}
I20260926 19:30:06.657953 14527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.646597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:42559 (local address 127.0.0.1:58176)
0926 19:30:06.649687 (+ 3090us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.649711 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.649942 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.650951 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.650957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.650986 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.651358 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.651367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.653219 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.653224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.654931 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.654944 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.657065 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.657080 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.657245 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.657274 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.657399 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.657549 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2899,"spinlock_wait_cycles":36736,"thread_start_us":410,"threads_started":1}
I20260926 19:30:06.658203 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.647154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.1.39.1:42559)
0926 19:30:06.649024 (+ 1870us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.649036 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.649819 (+ 783us) server_negotiation.cc:407] Connection header received
0926 19:30:06.650157 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.650169 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.650499 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.650682 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.651516 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.653083 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.655124 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.656544 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.656666 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.656900 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.657685 (+ 785us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.657866 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1668,"thread_start_us":1500,"threads_started":1}
I20260926 19:30:06.659061 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.646557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.1.39.2:36211)
0926 19:30:06.648356 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.648370 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.648404 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:30:06.648521 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.648531 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.648793 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.649895 (+ 1102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.650130 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.651685 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.654627 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.656033 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.658246 (+ 2213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.658491 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.658583 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.658748 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":1414,"threads_started":1}
I20260926 19:30:06.659293 14531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.648059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38991 (local address 127.0.0.1:48364)
0926 19:30:06.650942 (+ 2883us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.650958 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.651082 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.651610 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.651617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.651645 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.652041 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.652054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.654698 (+ 2644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.654703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.656262 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.656280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.658576 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.658581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.658694 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.658715 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.658829 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.658961 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2754,"thread_start_us":323,"threads_started":1}
I20260926 19:30:06.659484 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.648526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48364 (local address 127.1.39.3:38991)
0926 19:30:06.650808 (+ 2282us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.650816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.651047 (+ 231us) server_negotiation.cc:407] Connection header received
0926 19:30:06.651236 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.651243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.651429 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.651593 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.652255 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.654558 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.656456 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.658034 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.658187 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.658431 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.658884 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.659037 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2123,"thread_start_us":1982,"threads_started":1}
I20260926 19:30:06.662427 14305 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 (DEFAULT_TABLE table=test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:06.662427 14161 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 (DEFAULT_TABLE table=test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:06.662773 14440 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 (DEFAULT_TABLE table=test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:06.663920 14440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6180f57f474616b643bb9b9cd64fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:06.663919 14161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6180f57f474616b643bb9b9cd64fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:06.663934 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6180f57f474616b643bb9b9cd64fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:06.676929 14533 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Bootstrap starting.
I20260926 19:30:06.680589 14533 tablet_bootstrap.cc:654] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:06.681799 14533 log.cc:824] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:06.682034 14535 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Bootstrap starting.
I20260926 19:30:06.684222 14534 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Bootstrap starting.
I20260926 19:30:06.686017 14533 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: No bootstrap required, opened a new log
I20260926 19:30:06.686163 14535 tablet_bootstrap.cc:654] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:06.686347 14533 ts_tablet_manager.cc:1397] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260926 19:30:06.687498 14535 log.cc:824] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:06.688091 14534 tablet_bootstrap.cc:654] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:06.689704 14534 log.cc:824] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:06.691869 14535 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: No bootstrap required, opened a new log
I20260926 19:30:06.692168 14535 ts_tablet_manager.cc:1397] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 19:30:06.692633 14534 tablet_bootstrap.cc:492] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: No bootstrap required, opened a new log
I20260926 19:30:06.692884 14534 ts_tablet_manager.cc:1397] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 19:30:06.694738 14533 raft_consensus.cc:348] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.695086 14533 raft_consensus.cc:374] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:06.695240 14533 raft_consensus.cc:729] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3518f1b04a64028af0403675a1458f5, State: Initialized, Role: FOLLOWER
I20260926 19:30:06.695844 14533 consensus_queue.cc:260] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.697005 14505 heartbeater.cc:503] Master 127.1.39.62:36285 was elected leader, sending a full tablet report...
I20260926 19:30:06.697558 14533 ts_tablet_manager.cc:1428] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 19:30:06.702678 14535 raft_consensus.cc:348] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.703037 14535 raft_consensus.cc:374] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:06.702826 14534 raft_consensus.cc:348] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.703169 14535 raft_consensus.cc:729] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615780e398d545d48484f8b2250f6fe7, State: Initialized, Role: FOLLOWER
I20260926 19:30:06.703297 14534 raft_consensus.cc:374] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:06.703433 14534 raft_consensus.cc:729] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b564cfe6f9734b139105287e27e4089c, State: Initialized, Role: FOLLOWER
I20260926 19:30:06.703819 14535 consensus_queue.cc:260] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.704020 14534 consensus_queue.cc:260] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.705029 14226 heartbeater.cc:503] Master 127.1.39.62:36285 was elected leader, sending a full tablet report...
I20260926 19:30:06.705595 14535 ts_tablet_manager.cc:1428] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260926 19:30:06.705709 14534 ts_tablet_manager.cc:1428] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
W20260926 19:30:06.796816 14506 tablet.cc:2369] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:06.809350 14372 tablet.cc:2369] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:06.909789 14227 tablet.cc:2369] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:06.933039 14541 raft_consensus.cc:482] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:06.933394 14541 raft_consensus.cc:504] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.935124 14541 leader_election.cc:288] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211), c3518f1b04a64028af0403675a1458f5 (127.1.39.3:38991)
I20260926 19:30:06.944394 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.935679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:49321 (local address 127.1.39.2:36211)
0926 19:30:06.935888 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.935892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.936644 (+ 752us) server_negotiation.cc:407] Connection header received
0926 19:30:06.936810 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.936816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.936895 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.936997 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.937930 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.939012 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.942140 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.943372 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.943515 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.943629 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.943977 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.944142 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 19:30:06.944394 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.935680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:33365 (local address 127.1.39.3:38991)
0926 19:30:06.936070 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.936078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.936618 (+ 540us) server_negotiation.cc:407] Connection header received
0926 19:30:06.936810 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.936816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.936895 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.936997 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.937929 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.939006 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.942354 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.943531 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.943684 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.943772 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.944058 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.944175 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260926 19:30:06.944834 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.935636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:36211 (local address 127.1.39.1:49321)
0926 19:30:06.936533 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.936549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.936671 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.937031 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.937037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.937082 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.937776 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.937785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.939147 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.939152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.941943 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.941961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.943590 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.943595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.943799 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.943841 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.943942 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.944085 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":340,"spinlock_wait_cycles":57856,"thread_start_us":465,"threads_started":1}
I20260926 19:30:06.944941 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.935615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38991 (local address 127.1.39.1:33365)
0926 19:30:06.936456 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.936483 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.936661 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.937031 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.937037 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.937081 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.937755 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.937776 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.939149 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.939152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.941960 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.941969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.943746 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.943752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.943909 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.943944 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.944017 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.944129 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":245,"spinlock_wait_cycles":16000,"thread_start_us":638,"threads_started":1}
I20260926 19:30:06.944886 14539 raft_consensus.cc:482] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:06.945187 14539 raft_consensus.cc:504] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.946064 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "b564cfe6f9734b139105287e27e4089c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615780e398d545d48484f8b2250f6fe7" is_pre_election: true
I20260926 19:30:06.946529 14325 raft_consensus.cc:2454] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b564cfe6f9734b139105287e27e4089c in term 0.
I20260926 19:30:06.946753 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "b564cfe6f9734b139105287e27e4089c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3518f1b04a64028af0403675a1458f5" is_pre_election: true
I20260926 19:30:06.947077 14539 leader_election.cc:288] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211), b564cfe6f9734b139105287e27e4089c (127.1.39.1:42559)
I20260926 19:30:06.947183 14117 leader_election.cc:302] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615780e398d545d48484f8b2250f6fe7, b564cfe6f9734b139105287e27e4089c; no voters:
I20260926 19:30:06.947178 14460 raft_consensus.cc:2454] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b564cfe6f9734b139105287e27e4089c in term 0.
I20260926 19:30:06.947561 14541 raft_consensus.cc:2781] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:30:06.947708 14541 raft_consensus.cc:482] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:30:06.947826 14541 raft_consensus.cc:3037] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:06.951171 14541 raft_consensus.cc:504] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.951898 14541 leader_election.cc:288] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [CANDIDATE]: Term 1 election: Requested vote from peers 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211), c3518f1b04a64028af0403675a1458f5 (127.1.39.3:38991)
I20260926 19:30:06.952315 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "b564cfe6f9734b139105287e27e4089c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615780e398d545d48484f8b2250f6fe7"
I20260926 19:30:06.952519 14325 raft_consensus.cc:3037] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:06.952629 14460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "b564cfe6f9734b139105287e27e4089c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3518f1b04a64028af0403675a1458f5"
I20260926 19:30:06.952827 14460 raft_consensus.cc:3037] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:06.955101 14509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.947621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:36211 (local address 127.1.39.3:51785)
0926 19:30:06.947870 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.947889 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.947999 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.948882 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.948889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.948922 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.949129 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.949141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.950669 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.950673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.952485 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.952498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.954329 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.954335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.954527 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.954587 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.954658 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.954793 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 19:30:06.955564 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.948052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:45635 (local address 127.1.39.1:42559)
0926 19:30:06.948239 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.948245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.948505 (+ 260us) server_negotiation.cc:407] Connection header received
0926 19:30:06.948762 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.948767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.948858 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.948929 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.949688 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.950656 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.953196 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.954442 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.954537 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.954633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.955221 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.955348 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:30:06.955876 14325 raft_consensus.cc:2454] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b564cfe6f9734b139105287e27e4089c in term 1.
I20260926 19:30:06.955978 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.948364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:51785 (local address 127.1.39.2:36211)
0926 19:30:06.948556 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:06.948562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:06.948579 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:06.948638 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:06.948643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:06.948736 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:06.949266 (+ 530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:06.949426 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.950539 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.952748 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.954136 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.955407 (+ 1271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.955510 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:06.955564 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 19:30:06.955685 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:30:06.956314 14117 leader_election.cc:302] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615780e398d545d48484f8b2250f6fe7, b564cfe6f9734b139105287e27e4089c; no voters:
I20260926 19:30:06.956452 14325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "c3518f1b04a64028af0403675a1458f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "615780e398d545d48484f8b2250f6fe7" is_pre_election: true
I20260926 19:30:06.956048 14460 raft_consensus.cc:2454] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b564cfe6f9734b139105287e27e4089c in term 1.
I20260926 19:30:06.956604 14541 raft_consensus.cc:2781] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:06.956808 14325 raft_consensus.cc:2379] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3518f1b04a64028af0403675a1458f5 in current term 1: Already voted for candidate b564cfe6f9734b139105287e27e4089c in this term.
I20260926 19:30:06.956854 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:06.947859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:42559 (local address 127.1.39.3:45635)
0926 19:30:06.948417 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:06.948435 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:06.948604 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:06.949124 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:06.949130 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:06.949160 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:06.949521 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.949533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.950810 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.950815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.952979 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:06.952996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:06.954825 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:06.954831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:06.955025 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:06.955058 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:30:06.955768 (+ 710us) client_negotiation.cc:240] Negotiation successful
0926 19:30:06.955912 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":1,"thread_start_us":560,"threads_started":1}
I20260926 19:30:06.957988 14181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6180f57f474616b643bb9b9cd64fc9" candidate_uuid: "c3518f1b04a64028af0403675a1458f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b564cfe6f9734b139105287e27e4089c" is_pre_election: true
I20260926 19:30:06.957538 14541 raft_consensus.cc:686] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 1 LEADER]: Becoming Leader. State: Replica: b564cfe6f9734b139105287e27e4089c, State: Running, Role: LEADER
I20260926 19:30:06.958841 14541 consensus_queue.cc:237] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } }
I20260926 19:30:06.962718 14393 leader_election.cc:302] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3518f1b04a64028af0403675a1458f5; no voters: 615780e398d545d48484f8b2250f6fe7, b564cfe6f9734b139105287e27e4089c
I20260926 19:30:06.963097 14539 raft_consensus.cc:2726] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:30:06.963611 14028 catalog_manager.cc:5495] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c reported cstate change: term changed from 0 to 1, leader changed from <none> to b564cfe6f9734b139105287e27e4089c (127.1.39.1). New cstate: current_term: 1 leader_uuid: "b564cfe6f9734b139105287e27e4089c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b564cfe6f9734b139105287e27e4089c" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 } health_report { overall_health: UNKNOWN } } }
I20260926 19:30:07.027129 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:07.021880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.1.39.1:42559)
0926 19:30:07.022084 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:07.022089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:07.022127 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:30:07.022364 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:07.022368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:07.022458 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:07.022608 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:30:07.022906 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.023699 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.025022 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.025506 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.025603 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:07.026030 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:30:07.026529 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:30:07.026655 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:07.026734 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 19:30:07.026796 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:30:07.035010 14325 raft_consensus.cc:1261] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 1 FOLLOWER]: Refusing update from remote peer b564cfe6f9734b139105287e27e4089c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:07.035115 14460 raft_consensus.cc:1261] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 1 FOLLOWER]: Refusing update from remote peer b564cfe6f9734b139105287e27e4089c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:07.035928 14545 consensus_queue.cc:1035] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [LEADER]: Connected to new peer: Peer: permanent_uuid: "615780e398d545d48484f8b2250f6fe7" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 36211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:07.036254 14541 consensus_queue.cc:1035] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3518f1b04a64028af0403675a1458f5" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:07.046555 14548 mvcc.cc:205] Tried to move back new op lower bound from 7333687324805431296 to 7333687324506759168. Current Snapshot: MvccSnapshot[applied={T|T < 7333687324805431296}]
I20260926 19:30:07.047223 14550 mvcc.cc:205] Tried to move back new op lower bound from 7333687324805431296 to 7333687324506759168. Current Snapshot: MvccSnapshot[applied={T|T < 7333687324805431296}]
I20260926 19:30:07.064227 14029 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:07.064522 14029 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:07.070462 14029 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecae024a78d44207b16db05fe6f64135: Sending DeleteTablet for 3 replicas of tablet 7f6180f57f474616b643bb9b9cd64fc9
I20260926 19:30:07.071681 14305 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:07 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:33416
I20260926 19:30:07.071827 14440 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:07 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:48364
I20260926 19:30:07.071974 14161 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6180f57f474616b643bb9b9cd64fc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:07 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58176
I20260926 19:30:07.074524 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14066
I20260926 19:30:07.081096 14558 tablet_replica.cc:330] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: stopping tablet replica
I20260926 19:30:07.081619 14558 raft_consensus.cc:2229] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:07.082114 14558 raft_consensus.cc:2258] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:07.084896 14557 tablet_replica.cc:330] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: stopping tablet replica
I20260926 19:30:07.085737 14559 tablet_replica.cc:330] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: stopping tablet replica
I20260926 19:30:07.086151 14559 raft_consensus.cc:2229] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:30:07.086380 14557 raft_consensus.cc:2229] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:07.086479 14558 ts_tablet_manager.cc:1905] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:07.086874 14557 raft_consensus.cc:2258] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:07.087047 14559 raft_consensus.cc:2258] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:07.088816 14557 ts_tablet_manager.cc:1905] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:07.090929 14559 ts_tablet_manager.cc:1905] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:07.094130 14558 ts_tablet_manager.cc:1918] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:07.094306 14558 log.cc:1196] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/7f6180f57f474616b643bb9b9cd64fc9
I20260926 19:30:07.094856 14558 ts_tablet_manager.cc:1939] T 7f6180f57f474616b643bb9b9cd64fc9 P c3518f1b04a64028af0403675a1458f5: Deleting consensus metadata
I20260926 19:30:07.095991 14557 ts_tablet_manager.cc:1918] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:07.096210 14557 log.cc:1196] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/7f6180f57f474616b643bb9b9cd64fc9
I20260926 19:30:07.096860 14014 catalog_manager.cc:4841] TS c3518f1b04a64028af0403675a1458f5 (127.1.39.3:38991): tablet 7f6180f57f474616b643bb9b9cd64fc9 (table test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]) successfully deleted
I20260926 19:30:07.097124 14557 ts_tablet_manager.cc:1939] T 7f6180f57f474616b643bb9b9cd64fc9 P 615780e398d545d48484f8b2250f6fe7: Deleting consensus metadata
I20260926 19:30:07.097252 14559 ts_tablet_manager.cc:1918] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:07.097478 14559 log.cc:1196] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/7f6180f57f474616b643bb9b9cd64fc9
I20260926 19:30:07.098004 14559 ts_tablet_manager.cc:1939] T 7f6180f57f474616b643bb9b9cd64fc9 P b564cfe6f9734b139105287e27e4089c: Deleting consensus metadata
I20260926 19:30:07.098794 14016 catalog_manager.cc:4841] TS 615780e398d545d48484f8b2250f6fe7 (127.1.39.2:36211): tablet 7f6180f57f474616b643bb9b9cd64fc9 (table test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]) successfully deleted
I20260926 19:30:07.099514 14014 catalog_manager.cc:4841] TS b564cfe6f9734b139105287e27e4089c (127.1.39.1:42559): tablet 7f6180f57f474616b643bb9b9cd64fc9 (table test-table [id=c7b2dc2b134f490f97eee8f74c740e2b]) successfully deleted
I20260926 19:30:07.101477 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:07.075910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.1.39.1:42559)
0926 19:30:07.076139 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:07.076148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:07.077907 (+ 1759us) server_negotiation.cc:407] Connection header received
0926 19:30:07.078250 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:07.078258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:07.078366 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:07.078454 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:07.080828 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.082007 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.084458 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.085147 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.085255 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:07.089006 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:07.089019 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:07.089039 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:07.089080 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:07.093424 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.099120 (+ 5696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.099129 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.099133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.099286 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.099812 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.099817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.099821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.100228 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:07.100549 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:07.100875 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 19:30:07.101123 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:30:07.102137 14221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:07.202432 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14066
I20260926 19:30:07.220552 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14230
I20260926 19:30:07.233173 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:07.221393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.1.39.2:36211)
0926 19:30:07.221552 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:07.221556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:07.221573 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:07.221759 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:07.221763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:07.221831 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:07.221913 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:07.223274 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.223934 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.225119 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.225565 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.225657 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:07.228169 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:07.228181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:07.228197 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:07.228228 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:07.230986 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.231573 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.231579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.231581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.231705 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.231989 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.231991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.231993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.232328 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:07.232574 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:07.232756 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 19:30:07.232920 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:30:07.233826 14365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:07.324496 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14230
I20260926 19:30:07.341074 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14375
I20260926 19:30:07.354475 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:07.341845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.1.39.3:38991)
0926 19:30:07.342021 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:07.342028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:07.342103 (+ 75us) server_negotiation.cc:407] Connection header received
0926 19:30:07.342305 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:07.342310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:07.342383 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:07.342492 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:07.343796 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.344522 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.346017 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:07.346403 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:07.346532 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:07.348975 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:07.348988 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:07.349005 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:07.349041 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:07.351749 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.352390 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.352397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.352399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.352533 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:07.352899 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:07.352906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:07.352909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:07.353257 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:07.353593 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:07.353894 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 19:30:07.354114 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:30:07.355209 14500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:07.444972 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14375
I20260926 19:30:07.461496 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 13996
I20260926 19:30:07.462704 14059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:07.585570 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 13996
I20260926 19:30:07.608347 1180 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 19:30:07.943523 1180 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 19:30:07.943768 1180 mini_ranger.cc:66] Stopping Ranger...
I20260926 19:30:08.289119 1180 mini_ranger.cc:68] Stopped Ranger
2026-09-26 19:30:08.289 UTC [13048] LOG: received smart shutdown request
2026-09-26 19:30:08.292 UTC [13048] LOG: background worker "logical replication launcher" (PID 13057) exited with exit code 1
2026-09-26 19:30:08.292 UTC [13052] LOG: shutting down
2026-09-26 19:30:08.483 UTC [13048] LOG: database system is shut down
2026-09-26T19:30:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54932 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:30:08 dist-test-slave-9wb0 krb5kdc[14573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:30:08 dist-test-slave-9wb0 krb5kdc[14573](info): set up 2 sockets
Sep 26 19:30:08 dist-test-slave-9wb0 krb5kdc[14573](info): commencing operation
krb5kdc: starting...
W20260926 19:30:09.729789 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:30:09 dist-test-slave-9wb0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:30:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:30:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:30:09.976454 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:39087
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:10.308538 14586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:10.308984 14586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:10.309152 14586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:10.317009 14586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:10.317116 14586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:10.317162 14586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:10.317209 14586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:10.317252 14586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:10.327143 14586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:39087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:10.328421 14586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:10.329743 14586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:10.341794 14591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:10.341801 14594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:10.341832 14592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:10.342641 14586 server_base.cc:1034] running on GCE node
I20260926 19:30:10.343436 14586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:10.344869 14586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:10.346112 14586 hybrid_clock.cc:648] HybridClock initialized: now 1790451010346058 us; error 43 us; skew 500 ppm
I20260926 19:30:10.346513 14586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:10 dist-test-slave-9wb0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:10.349699 14586 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:10.351046 14586 webserver.cc:460] Webserver started at http://127.1.39.62:37999/ using document root <none> and password file <none>
I20260926 19:30:10.351697 14586 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:10.351794 14586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:10.352028 14586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:10.354998 14586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "12bbd123587f4f989ee71f183268c27c"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "c598d6f9d63dfe8d028020a619033fa0"
server_key_iv: "b6e018d401c98fd9e00bb00674ca47fb"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.355755 14586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "12bbd123587f4f989ee71f183268c27c"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "c598d6f9d63dfe8d028020a619033fa0"
server_key_iv: "b6e018d401c98fd9e00bb00674ca47fb"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.360860 14586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:30:10.364679 14601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:10.366191 14586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:10.366377 14586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "12bbd123587f4f989ee71f183268c27c"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "c598d6f9d63dfe8d028020a619033fa0"
server_key_iv: "b6e018d401c98fd9e00bb00674ca47fb"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.366577 14586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:10.381479 14586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:10.385828 14586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:10.386122 14586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:10.405099 14586 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:39087
I20260926 19:30:10.405185 14652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:39087 every 8 connection(s)
I20260926 19:30:10.406502 14586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:10.410410 14653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:10.410465 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14586
I20260926 19:30:10.410706 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:30:10.411520 1180 external_mini_cluster.cc:1428] Setting key efb2fcd3fc17d4a728aa0a8c3329158a
I20260926 19:30:10.420733 14653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.39.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.61@KRBTEST.COM" created.
Sep 26 19:30:10 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:10.446139 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.425482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.1.39.62:39087)
0926 19:30:10.426179 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:10.426193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:10.426241 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:30:10.427394 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:10.427400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:10.427611 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:10.427805 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:10.429033 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.430300 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.431670 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.432205 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.432403 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.438797 (+ 6394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:10.438806 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:10.438819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:10.438850 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:10.441738 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.442660 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.442667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.442669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.442811 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.443283 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.443286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.443288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.443811 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:10.443968 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:10.444670 (+ 702us) server_negotiation.cc:299] Negotiation successful
0926 19:30:10.445157 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":477,"thread_start_us":314,"threads_started":1}
I20260926 19:30:10.446171 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.424096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39087 (local address 127.0.0.1:55990)
0926 19:30:10.424918 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.424961 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:10.427032 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:10.427818 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:10.427824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:10.428517 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:10.428830 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.428847 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.430462 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.430471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.431531 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.431543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.432430 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.432436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.432642 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.433394 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:10.433419 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:10.438518 (+ 5099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:10.441884 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.441900 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.441915 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.442536 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.442922 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.442929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.442935 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.443167 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.443952 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:10.443963 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:10.444226 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 19:30:10.444509 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 19:30:10.444984 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"thread_start_us":368,"threads_started":1}
I20260926 19:30:10.449241 14653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:30:10.450408 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.450038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41192)
0926 19:30:10.450210 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.450308 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 19:30:10.450701 14602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
I20260926 19:30:10.452312 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.451947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41196)
0926 19:30:10.452113 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.452187 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 19:30:10.452526 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44073: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
I20260926 19:30:10.469420 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:44073
--webserver_interface=127.1.39.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:10.501837 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } attempt: 1
I20260926 19:30:10.502954 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.502522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41204)
0926 19:30:10.502716 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.502829 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:30:10.504089 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.503709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41216)
0926 19:30:10.503867 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.503946 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 19:30:10.504277 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44073: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
I20260926 19:30:10.572566 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } attempt: 2
I20260926 19:30:10.573741 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.573242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41228)
0926 19:30:10.573505 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.573598 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 19:30:10.574891 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.574487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41232)
0926 19:30:10.574694 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.574759 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 19:30:10.575073 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44073: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
I20260926 19:30:10.666370 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } attempt: 3
I20260926 19:30:10.667698 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.667209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41246)
0926 19:30:10.667413 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.667547 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 19:30:10.668784 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.668372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41258)
0926 19:30:10.668570 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.668640 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 19:30:10.669011 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44073: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
W20260926 19:30:10.799118 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:10.799474 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:10.799649 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:10.806963 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:10.807121 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:10.807241 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:10.807350 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:10.807440 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:10.812240 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } attempt: 4
I20260926 19:30:10.813589 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.812965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41262)
0926 19:30:10.813274 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.813443 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260926 19:30:10.814654 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.814260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41270)
0926 19:30:10.814428 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.814528 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 19:30:10.814849 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44073: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44073: connect: Connection refused (error 111)
I20260926 19:30:10.818285 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.39.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:10.819609 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:10.820899 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:10.832307 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:10.832522 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:10.832747 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:10.833554 14660 server_base.cc:1034] running on GCE node
I20260926 19:30:10.834349 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:10.835708 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:10.836937 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790451010836898 us; error 60 us; skew 500 ppm
I20260926 19:30:10.837271 14660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:10 dist-test-slave-9wb0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:10.840462 14660 init.cc:377] Logged in from keytab as kudu/127.1.39.61@KRBTEST.COM (short username kudu)
I20260926 19:30:10.842006 14660 webserver.cc:460] Webserver started at http://127.1.39.61:42137/ using document root <none> and password file <none>
I20260926 19:30:10.842643 14660 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:10.842748 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:10.842983 14660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:10.845890 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data/instance:
uuid: "e0e98fd7622947b1bc3aedca474be6ea"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "652ea0b91a317b95aad37b1ed48e3e41"
server_key_iv: "43fca9da330a4342793a6191f7677326"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.846626 14660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal/instance:
uuid: "e0e98fd7622947b1bc3aedca474be6ea"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "652ea0b91a317b95aad37b1ed48e3e41"
server_key_iv: "43fca9da330a4342793a6191f7677326"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.851682 14660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:30:10.855059 14675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:10.856432 14660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:30:10.856568 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "e0e98fd7622947b1bc3aedca474be6ea"
format_stamp: "Formatted at 2026-09-26 19:30:10 on dist-test-slave-9wb0"
server_key: "652ea0b91a317b95aad37b1ed48e3e41"
server_key_iv: "43fca9da330a4342793a6191f7677326"
server_key_version: "encryptionkey@0"
I20260926 19:30:10.856724 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:10.885766 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:10.890269 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:10.890516 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:10.909775 14660 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.61:44073
I20260926 19:30:10.909822 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.61:44073 every 8 connection(s)
I20260926 19:30:10.911221 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
I20260926 19:30:10.914119 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14660
I20260926 19:30:10.914340 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-1/wal/instance
I20260926 19:30:10.914916 1180 external_mini_cluster.cc:1428] Setting key 4f048a93301b51bf80f95134fea4146b
I20260926 19:30:10.915611 14727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:10.926128 14727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.39.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.60@KRBTEST.COM" created.
Sep 26 19:30:10 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:10.949252 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.928994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.1.39.62:39087)
0926 19:30:10.929194 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:10.929199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:10.930554 (+ 1355us) server_negotiation.cc:407] Connection header received
0926 19:30:10.932540 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:10.932547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:10.932643 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:10.932767 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:10.934424 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.935170 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.937069 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.937619 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.937770 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.942067 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:10.942074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:10.942080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:10.942131 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:10.945903 (+ 3772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.946702 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.946711 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.946717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.946867 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.947257 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.947264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.947270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.947688 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:10.947843 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:10.948528 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 19:30:10.948798 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:30:10.950067 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.929518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39087 (local address 127.0.0.1:55992)
0926 19:30:10.930347 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.930392 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:10.932240 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:10.933014 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:10.933023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:10.933646 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:10.934238 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.934256 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.935358 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.935377 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.936902 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.936916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.937873 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.937881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.938147 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.938804 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:10.938857 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:10.941811 (+ 2954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:10.946072 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.946078 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.946094 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.946545 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.946972 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.946976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.946978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.947142 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.947810 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:10.947816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:10.948145 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 19:30:10.948370 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 19:30:10.948761 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":354,"threads_started":1}
I20260926 19:30:10.951874 14727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:30:10 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:30:10.966936 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.952635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41276)
0926 19:30:10.952788 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.952803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:10.952893 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:10.954103 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:10.954110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:10.954506 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:10.954633 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.954641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.956042 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.956047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.956895 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:10.956905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.957791 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.957797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:10.958002 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.958610 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:10.958627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:10.960732 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:10.964054 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.964061 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.964067 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.964729 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.965133 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:10.965138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:10.965142 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:10.965333 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:10.965940 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:10.965947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:10.966119 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 19:30:10.966259 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 19:30:10.966542 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 19:30:10.967024 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.952947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.1.39.61:44073)
0926 19:30:10.953644 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:10.953650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:10.953674 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:30:10.953718 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:10.953724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:10.953887 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:10.954054 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:10.954780 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.955885 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.957027 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:10.957634 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:10.957774 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:10.960935 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:10.960946 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:10.960958 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:10.960984 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:10.963866 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.964865 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.964891 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.964893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.965004 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:10.965443 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:10.965448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:10.965451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:10.965798 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:10.966012 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:10.966396 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 19:30:10.966658 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":548,"thread_start_us":426,"threads_started":1}
I20260926 19:30:10.969652 14727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:30:10.970832 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.970357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51004)
0926 19:30:10.970509 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.970640 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 19:30:10.971053 14676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
I20260926 19:30:10.972730 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:10.972330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51016)
0926 19:30:10.972491 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:10.972592 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 19:30:10.972707 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:44873
--webserver_interface=127.1.39.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:43959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:10.973006 14727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.022326 14727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 1
I20260926 19:30:11.023604 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.023128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51020)
0926 19:30:11.023329 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.023433 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 19:30:11.024550 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.024222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51022)
0926 19:30:11.024384 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.024449 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:30:11.024735 14727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.092950 14727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 2
I20260926 19:30:11.094208 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.093740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51038)
0926 19:30:11.093947 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.094053 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 19:30:11.095173 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.094840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51040)
0926 19:30:11.095000 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.095064 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 19:30:11.095361 14727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.114156 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } attempt: 5
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:30:11.129379 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.115007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:41284)
0926 19:30:11.115219 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.115248 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.115357 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.115775 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.115778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.116202 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.116487 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.116499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.118310 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.118314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.119178 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.119198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.120116 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.120120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.120267 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.121014 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.121032 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.123128 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.127014 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.127018 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.127021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.127381 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.127811 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.127814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.127816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.127939 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.128630 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.128633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.128760 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.128882 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.129079 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 19:30:11.129746 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.115094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.1.39.61:44073)
0926 19:30:11.115290 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.115297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.115326 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:30:11.115504 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.115512 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.115611 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.115732 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.116679 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.118122 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.119326 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.119948 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.120076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.123346 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.123354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.123361 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.123409 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.126869 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.127506 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.127514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.127519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.127686 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.128050 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.128058 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.128062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.128501 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.128676 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.129046 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.129350 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:30:11.130834 14653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:30:11.131795 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.131428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51044)
0926 19:30:11.131584 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.131667 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 19:30:11.132650 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.132347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51054)
0926 19:30:11.132484 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.132564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:30:11.132843 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.184103 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 1
I20260926 19:30:11.185390 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.184855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51068)
0926 19:30:11.185057 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.185160 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 19:30:11.186358 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.186035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51070)
0926 19:30:11.186181 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.186257 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 19:30:11.186576 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.186607 14727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 3
I20260926 19:30:11.187763 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.187279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51086)
0926 19:30:11.187499 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.187597 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 19:30:11.188868 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.188410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51094)
0926 19:30:11.188642 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.188732 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 19:30:11.189074 14727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.254820 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 2
I20260926 19:30:11.255963 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.255482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51100)
0926 19:30:11.255721 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.255805 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 19:30:11.256899 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.256590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51102)
0926 19:30:11.256731 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.256803 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 19:30:11.257133 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
W20260926 19:30:11.316774 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:11.317118 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:11.317312 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:11.324431 14734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:11.324575 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:11.324692 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:11.324796 14734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:11.324882 14734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:11.332297 14727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 4
I20260926 19:30:11.333521 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.332987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51104)
0926 19:30:11.333221 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.333352 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:30:11.334450 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.334144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:51116)
0926 19:30:11.334276 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.334349 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:30:11.334659 14727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.335181 14734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:43959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:39087,127.1.39.61:44073,127.1.39.60:44873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.39.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:11.336493 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:11.337893 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:11.349244 14742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:11.349390 14739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:11.349220 14740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:11.350189 14734 server_base.cc:1034] running on GCE node
I20260926 19:30:11.350950 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:11.352335 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:11.353557 14734 hybrid_clock.cc:648] HybridClock initialized: now 1790451011353519 us; error 55 us; skew 500 ppm
I20260926 19:30:11.353919 14734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:11.356999 14734 init.cc:377] Logged in from keytab as kudu/127.1.39.60@KRBTEST.COM (short username kudu)
I20260926 19:30:11.358666 14734 webserver.cc:460] Webserver started at http://127.1.39.60:40733/ using document root <none> and password file <none>
I20260926 19:30:11.359396 14734 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:11.359541 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:11.359889 14734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:11.363020 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data/instance:
uuid: "7ec16a3b84c74a70aff59c11120bdce9"
format_stamp: "Formatted at 2026-09-26 19:30:11 on dist-test-slave-9wb0"
server_key: "525c39c6e0436972773f52507953704f"
server_key_iv: "c84ee63b630a43a1c420ebc4f7ae0d2b"
server_key_version: "encryptionkey@0"
I20260926 19:30:11.363343 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 3
I20260926 19:30:11.363924 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal/instance:
uuid: "7ec16a3b84c74a70aff59c11120bdce9"
format_stamp: "Formatted at 2026-09-26 19:30:11 on dist-test-slave-9wb0"
server_key: "525c39c6e0436972773f52507953704f"
server_key_iv: "c84ee63b630a43a1c420ebc4f7ae0d2b"
server_key_version: "encryptionkey@0"
I20260926 19:30:11.364451 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.363990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:43698)
0926 19:30:11.364218 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.364295 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 19:30:11.365486 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.365097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:43706)
0926 19:30:11.365267 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.365380 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:30:11.365680 14653 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:44873: Network error: Client connection negotiation failed: client connection to 127.1.39.60:44873: connect: Connection refused (error 111)
I20260926 19:30:11.369470 14734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 19:30:11.373086 14749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:11.374460 14734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:30:11.374666 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "7ec16a3b84c74a70aff59c11120bdce9"
format_stamp: "Formatted at 2026-09-26 19:30:11 on dist-test-slave-9wb0"
server_key: "525c39c6e0436972773f52507953704f"
server_key_iv: "c84ee63b630a43a1c420ebc4f7ae0d2b"
server_key_version: "encryptionkey@0"
I20260926 19:30:11.374871 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:11.398200 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:11.402446 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:11.402753 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:11.422928 14734 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.60:44873
I20260926 19:30:11.422991 14800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.60:44873 every 8 connection(s)
I20260926 19:30:11.424281 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
I20260926 19:30:11.427306 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14734
I20260926 19:30:11.427520 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790450801920901-1180-0/minicluster-data/master-2/wal/instance
I20260926 19:30:11.428172 1180 external_mini_cluster.cc:1428] Setting key 787613ecca6943585d15787a53795a65
I20260926 19:30:11.428426 14801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:11.438334 14801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:11.453408 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.430293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43716 (local address 127.1.39.60:44873)
0926 19:30:11.431124 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.431139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.431188 (+ 49us) server_negotiation.cc:407] Connection header received
0926 19:30:11.433337 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.433366 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.433883 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.434333 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.435740 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.437686 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.439204 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.439865 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.440071 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.444924 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.444931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.444941 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.444978 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.448726 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.449913 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.449923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.449932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.450153 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.450618 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.450622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.450624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.451171 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.451464 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.451935 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.452355 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":340,"threads_started":1}
I20260926 19:30:11.457504 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.440744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:39087 (local address 127.0.0.1:48454)
0926 19:30:11.441520 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.441560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.441738 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.442112 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.442119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.442790 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.443104 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.443139 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.444250 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.444256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.445882 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.445897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.446969 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.446981 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.447157 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.447799 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.447823 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.451084 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.454418 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.454428 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.454436 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.455035 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.455447 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.455452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.455455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.455642 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.456293 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.456301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.456492 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.456709 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.457047 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":377,"threads_started":1}
I20260926 19:30:11.457499 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.440727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.1.39.62:39087)
0926 19:30:11.440935 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.440944 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.441653 (+ 709us) server_negotiation.cc:407] Connection header received
0926 19:30:11.441864 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.441870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.441941 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.442054 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.443294 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.444078 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.446036 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.446742 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.446836 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.451292 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.451297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.451300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.451334 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.454270 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.455183 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.455188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.455190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.455321 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.455797 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.455805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.455810 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.456138 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.456295 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.456844 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.457074 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:30:11.458933 14801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:11.474411 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.459649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44073 (local address 127.0.0.1:33990)
0926 19:30:11.459816 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.459834 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.459928 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.460330 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.460333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.460721 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.460898 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.460910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.461973 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.461979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.463128 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.463144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.463963 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.463970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.464198 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.464708 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.464734 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.467184 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.471688 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.471691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.471694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.472069 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.472528 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.472534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.472538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.472761 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.473450 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.473454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.473742 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.473906 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.474146 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 19:30:11.474722 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.459822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.1.39.61:44073)
0926 19:30:11.460028 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.460034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.460052 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:30:11.460128 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.460133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.460205 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.460304 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.461072 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.461833 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.463297 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.463813 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.463904 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.467414 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.467422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.467428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.467475 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.471555 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.472191 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.472198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.472204 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.472384 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.472916 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.472920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.472922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.473286 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.473499 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.474050 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.474276 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:30:11.475814 14801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:30:11.476052 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.456672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.1.39.62:39087)
0926 19:30:11.457627 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.457635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.457664 (+ 29us) server_negotiation.cc:407] Connection header received
0926 19:30:11.457721 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.457728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.457800 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.457944 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.459507 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.460626 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.462221 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.462624 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.462714 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.468477 (+ 5763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.468482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.468486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.468520 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.472341 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.473225 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.473232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.473235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.473414 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.473940 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.473946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.473949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.474321 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.474630 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.475441 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.475673 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":586,"threads_started":1}
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451011, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:30:11.489820 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.476922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43732 (local address 127.1.39.60:44873)
0926 19:30:11.477092 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.477096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.477112 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:30:11.477150 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.477153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.477207 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.477331 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.478091 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.478930 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.480253 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.480789 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.480872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.484335 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.484339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.484342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.484369 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.487232 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.487963 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.487967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.487969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.488090 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.488455 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.488463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.488467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.488727 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.488875 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.489194 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.489454 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:30:11.489993 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.476565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:43732)
0926 19:30:11.476744 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.476768 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.476878 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.477361 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.477367 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.477715 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.477883 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.477895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.479081 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.479088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.480127 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.480139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.480939 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.480945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.481164 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.481691 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.481711 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.484166 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.487356 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.487360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.487364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.487843 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.488220 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.488225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.488229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.488356 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.488840 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.488844 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.488971 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.489124 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.489610 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":186}
I20260926 19:30:11.493148 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.478020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.1.39.61:44073)
0926 19:30:11.478191 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.478195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.478207 (+ 12us) server_negotiation.cc:407] Connection header received
0926 19:30:11.478239 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.478242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.478287 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.478384 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.479652 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.480428 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.482153 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.482551 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.482638 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.487401 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.487409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.487414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.487452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.490486 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.491289 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.491295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.491298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.491430 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.491782 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.491789 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.491792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.492067 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.492358 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.492607 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.492821 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 19:30:11.496003 14801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: Bootstrap starting.
I20260926 19:30:11.497946 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:30:11.499444 14801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:11.500486 14801 log.cc:824] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:11.503877 14801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: No bootstrap required, opened a new log
I20260926 19:30:11.514698 14801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.515213 14801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:11.515381 14801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec16a3b84c74a70aff59c11120bdce9, State: Initialized, Role: FOLLOWER
I20260926 19:30:11.516249 14801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.519006 14815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.519515 14815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.520267 14801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:11.524909 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.503283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34000 (local address 127.1.39.61:44073)
0926 19:30:11.503442 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.503448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.504414 (+ 966us) server_negotiation.cc:407] Connection header received
0926 19:30:11.504666 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.504676 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.504734 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.504838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.507689 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.508794 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.511155 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.511823 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.511922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.516369 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.516376 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.516380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.516425 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.520408 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.521246 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.521251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.521254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.521423 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.521933 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.521941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.521947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.522432 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.522753 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.524058 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.524400 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:30:11.525317 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.504227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.1.39.60:44873)
0926 19:30:11.504457 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.504463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.505836 (+ 1373us) server_negotiation.cc:407] Connection header received
0926 19:30:11.506055 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.506061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.506151 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.506260 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.510161 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.511334 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.513328 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.514134 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.514247 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.517859 (+ 3612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.517865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.517871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.517911 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.521820 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.522719 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.522725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.522728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.522903 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.523379 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.523384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.523387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.523742 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.524155 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.524675 (+ 520us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.524908 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:30:11.525949 14806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.505846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.1.39.62:39087)
0926 19:30:11.506049 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.506055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.506071 (+ 16us) server_negotiation.cc:407] Connection header received
0926 19:30:11.506125 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.506129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.506180 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.506285 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.508110 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.509249 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.511483 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.512166 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.512284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.516206 (+ 3922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.516214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.516220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.516264 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.522350 (+ 6086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.523263 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.523268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.523271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.523420 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.523874 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.523882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.523887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.524252 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.524477 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.525374 (+ 897us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.525623 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 19:30:11.536909 14826 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:30:11.537050 14826 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:30:11.534439 14734 master_runner.cc:186] Error getting master registration for 127.1.39.62:39087: OK, instance_id { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" instance_seqno: 1790451010399596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:30:11.542868 14734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:30:11.542917 14653 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 4
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:30:11.556548 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.543518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:43738)
0926 19:30:11.543701 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.543721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.543830 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.544196 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.544199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.544639 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.544862 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.544876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.546222 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.546227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.547124 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.547135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.547884 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.547890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.548072 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.548695 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.548711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.550768 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.553859 (+ 3091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.553865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.553869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.554233 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.554666 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.554674 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.554677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.554852 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.555573 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.555577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.555744 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.555903 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.556246 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 19:30:11.556656 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.543600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43738 (local address 127.1.39.60:44873)
0926 19:30:11.543752 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.543760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.543779 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:30:11.543962 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.543967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.544046 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.544143 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.545025 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.546080 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.547272 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.547737 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.547863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.550948 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.550953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.550956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.550988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.553729 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.554351 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.554358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.554361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.554517 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.554982 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.554986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.554989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.555369 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.555505 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.556034 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.556284 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:30:11.561226 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: Bootstrap starting.
I20260926 19:30:11.564533 14653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:11.565856 14653 log.cc:824] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:11.568533 14653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: No bootstrap required, opened a new log
I20260926 19:30:11.575918 14653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.576392 14653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:11.576577 14653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bbd123587f4f989ee71f183268c27c, State: Initialized, Role: FOLLOWER
I20260926 19:30:11.577373 14653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.578333 14829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.578833 14829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.579449 14653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:30:11.596264 14840 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:30:11.596495 14840 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:30:11.589103 14586 master_runner.cc:186] Error getting master registration for 127.1.39.61:44073: OK, instance_id { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" instance_seqno: 1790451010904229 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:30:11.601670 14586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:30:11.633913 14727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } attempt: 5
W20260926 19:30:11.638975 14607 tablet.cc:2369] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 19:30:11 dist-test-slave-9wb0 krb5kdc[14573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451010, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:30:11.652683 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.634606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:44873 (local address 127.0.0.1:43750)
0926 19:30:11.634878 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:11.634900 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:11.635021 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:11.635422 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:11.635428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:11.635961 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:11.636181 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.636195 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.637716 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.637722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.639212 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:11.639227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.640308 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.640314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:11.640518 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.641220 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:11.641246 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:11.645416 (+ 4170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:11.649708 (+ 4292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.649715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.649719 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.650278 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.650739 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:11.650745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:11.650748 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:11.650918 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:11.651604 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:11.651609 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:11.651787 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 19:30:11.651958 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 19:30:11.652284 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260926 19:30:11.652784 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:11.634695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.1.39.60:44873)
0926 19:30:11.634886 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:11.634892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:11.634961 (+ 69us) server_negotiation.cc:407] Connection header received
0926 19:30:11.635162 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:11.635168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:11.635256 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:11.635368 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:11.636357 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.637536 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.639457 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:11.640146 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:11.640297 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:11.645620 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:11.645628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:11.645632 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:11.645674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:11.649563 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.650409 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.650418 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.650423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.650613 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:11.651041 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:11.651046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:11.651050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:11.651430 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:11.651595 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:11.652090 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 19:30:11.652362 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 19:30:11.655097 14755 tablet.cc:2369] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:11.659173 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: Bootstrap starting.
I20260926 19:30:11.665565 14727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:11.666877 14727 log.cc:824] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:11.670348 14727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: No bootstrap required, opened a new log
I20260926 19:30:11.682891 14727 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.683346 14727 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:11.683460 14727 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e98fd7622947b1bc3aedca474be6ea, State: Initialized, Role: FOLLOWER
I20260926 19:30:11.684108 14727 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.685127 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.685635 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.686496 14727 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:11.706187 14660 master_runner.cc:263] No leader master found from master e0e98fd7622947b1bc3aedca474be6ea
I20260926 19:30:11.706357 14660 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 19:30:11.712162 14854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:30:11.712420 14854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:30:11.823051 14843 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:11.823400 14843 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.825125 14843 leader_election.cc:288] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12bbd123587f4f989ee71f183268c27c (127.1.39.62:39087), 7ec16a3b84c74a70aff59c11120bdce9 (127.1.39.60:44873)
I20260926 19:30:11.826524 14776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e98fd7622947b1bc3aedca474be6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec16a3b84c74a70aff59c11120bdce9" is_pre_election: true
I20260926 19:30:11.827198 14776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e98fd7622947b1bc3aedca474be6ea in term 0.
I20260926 19:30:11.828037 14676 leader_election.cc:302] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ec16a3b84c74a70aff59c11120bdce9, e0e98fd7622947b1bc3aedca474be6ea; no voters:
I20260926 19:30:11.829197 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e98fd7622947b1bc3aedca474be6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bbd123587f4f989ee71f183268c27c" is_pre_election: true
I20260926 19:30:11.829914 14628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e98fd7622947b1bc3aedca474be6ea in term 0.
I20260926 19:30:11.830410 14843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:30:11.830610 14843 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:30:11.830715 14843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:11.834544 14843 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.836566 14628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e98fd7622947b1bc3aedca474be6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bbd123587f4f989ee71f183268c27c"
I20260926 19:30:11.836835 14628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:11.840612 14628 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e98fd7622947b1bc3aedca474be6ea in term 1.
I20260926 19:30:11.845142 14677 leader_election.cc:302] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bbd123587f4f989ee71f183268c27c, e0e98fd7622947b1bc3aedca474be6ea; no voters:
I20260926 19:30:11.851617 14776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e0e98fd7622947b1bc3aedca474be6ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec16a3b84c74a70aff59c11120bdce9"
I20260926 19:30:11.851879 14776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:11.855845 14776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e98fd7622947b1bc3aedca474be6ea in term 1.
I20260926 19:30:11.856771 14843 leader_election.cc:288] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [CANDIDATE]: Term 1 election: Requested vote from peers 12bbd123587f4f989ee71f183268c27c (127.1.39.62:39087), 7ec16a3b84c74a70aff59c11120bdce9 (127.1.39.60:44873)
I20260926 19:30:11.857239 14843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:11.857767 14843 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [term 1 LEADER]: Becoming Leader. State: Replica: e0e98fd7622947b1bc3aedca474be6ea, State: Running, Role: LEADER
I20260926 19:30:11.858626 14843 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } }
I20260926 19:30:11.872611 14856 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e98fd7622947b1bc3aedca474be6ea. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.873253 14856 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:11.883464 14858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:11.890538 14858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 19:30:11.893054 14681 tablet.cc:2369] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:11.912149 14776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [term 1 FOLLOWER]: Refusing update from remote peer e0e98fd7622947b1bc3aedca474be6ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:11.913341 14856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:11.921352 14628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [term 1 FOLLOWER]: Refusing update from remote peer e0e98fd7622947b1bc3aedca474be6ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:11.922377 14856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:11.938783 14815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e98fd7622947b1bc3aedca474be6ea. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.939178 14815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.944561 14860 mvcc.cc:205] Tried to move back new op lower bound from 7333687344778936320 to 7333687344587481088. Current Snapshot: MvccSnapshot[applied={T|T < 7333687344778936320}]
I20260926 19:30:11.948048 14859 mvcc.cc:205] Tried to move back new op lower bound from 7333687344778936320 to 7333687344587481088. Current Snapshot: MvccSnapshot[applied={T|T < 7333687344778936320}]
I20260926 19:30:11.953446 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.954142 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:11.959640 14815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.959913 14815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.962073 14858 catalog_manager.cc:1347] Generated new cluster ID: 8da3e377064243b1a3eea1c29fd91a65
I20260926 19:30:11.962198 14858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:11.976734 14829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e98fd7622947b1bc3aedca474be6ea. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.977084 14829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:11.980055 14843 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.980366 14843 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:11.982177 14861 mvcc.cc:205] Tried to move back new op lower bound from 7333687344778936320 to 7333687344587481088. Current Snapshot: MvccSnapshot[applied={T|T < 7333687344778936320}]
I20260926 19:30:11.989811 14829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e98fd7622947b1bc3aedca474be6ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bbd123587f4f989ee71f183268c27c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 39087 } } peers { permanent_uuid: "e0e98fd7622947b1bc3aedca474be6ea" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44073 } } peers { permanent_uuid: "7ec16a3b84c74a70aff59c11120bdce9" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 44873 } } }
I20260926 19:30:11.990057 14829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:30:12.008826 14858 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:30:12.010730 14858 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:12.028265 14858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0e98fd7622947b1bc3aedca474be6ea: Generated new TSK 0
I20260926 19:30:12.029606 14858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:30:12.539075 14826 catalog_manager.cc:1259] Loaded cluster ID: 8da3e377064243b1a3eea1c29fd91a65
I20260926 19:30:12.539214 14826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: loading cluster ID for follower catalog manager: success
I20260926 19:30:12.541824 14826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: acquiring CA information for follower catalog manager: success
I20260926 19:30:12.543617 14826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ec16a3b84c74a70aff59c11120bdce9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:30:12.598477 14840 catalog_manager.cc:1259] Loaded cluster ID: 8da3e377064243b1a3eea1c29fd91a65
I20260926 19:30:12.598603 14840 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: loading cluster ID for follower catalog manager: success
I20260926 19:30:12.601366 14840 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: acquiring CA information for follower catalog manager: success
I20260926 19:30:12.603096 14840 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12bbd123587f4f989ee71f183268c27c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:30:12.619431 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14586
I20260926 19:30:12.620683 14648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:12.731467 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14586
I20260926 19:30:12.753542 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14660
I20260926 19:30:12.754619 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:12.863209 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14660
I20260926 19:30:12.885476 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14734
I20260926 19:30:12.886781 14796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:12.998580 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14734
2026-09-26T19:30:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4385 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:30:13 dist-test-slave-9wb0 krb5kdc[14877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:30:13 dist-test-slave-9wb0 krb5kdc[14877](info): set up 2 sockets
Sep 26 19:30:13 dist-test-slave-9wb0 krb5kdc[14877](info): commencing operation
krb5kdc: starting...
W20260926 19:30:14.109822 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:30:14 dist-test-slave-9wb0 krb5kdc[14877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:30:14.346951 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:35185
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:14.676990 14890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:14.677348 14890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:14.677522 14890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:14.684845 14890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:14.685000 14890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:14.685114 14890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:14.685213 14890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:14.685320 14890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:14.695261 14890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:14.696676 14890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:14.698040 14890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:14.709725 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:14.709869 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:14.709995 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:14.715013 14890 server_base.cc:1034] running on GCE node
I20260926 19:30:14.715903 14890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:14.717522 14890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:14.718744 14890 hybrid_clock.cc:648] HybridClock initialized: now 1790451014718694 us; error 31 us; skew 500 ppm
I20260926 19:30:14.719208 14890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:14 dist-test-slave-9wb0 krb5kdc[14877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451014, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:14.728775 14890 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:14.730557 14890 webserver.cc:460] Webserver started at http://127.1.39.62:46207/ using document root <none> and password file <none>
I20260926 19:30:14.731277 14890 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:14.731382 14890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:14.731650 14890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:14.734853 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "1d4e195d34384cb68be12716ccf3650c"
format_stamp: "Formatted at 2026-09-26 19:30:14 on dist-test-slave-9wb0"
server_key: "925be8929ef056cad52294f8f980073c"
server_key_iv: "32a997c2442bbdf8aad05ffa66476df2"
server_key_version: "encryptionkey@0"
I20260926 19:30:14.735605 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "1d4e195d34384cb68be12716ccf3650c"
format_stamp: "Formatted at 2026-09-26 19:30:14 on dist-test-slave-9wb0"
server_key: "925be8929ef056cad52294f8f980073c"
server_key_iv: "32a997c2442bbdf8aad05ffa66476df2"
server_key_version: "encryptionkey@0"
I20260926 19:30:14.740859 14890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260926 19:30:14.744468 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:14.745936 14890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:14.746133 14890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "1d4e195d34384cb68be12716ccf3650c"
format_stamp: "Formatted at 2026-09-26 19:30:14 on dist-test-slave-9wb0"
server_key: "925be8929ef056cad52294f8f980073c"
server_key_iv: "32a997c2442bbdf8aad05ffa66476df2"
server_key_version: "encryptionkey@0"
I20260926 19:30:14.746312 14890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:14.764801 14890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:14.769217 14890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:14.769526 14890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:14.789037 14890 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:35185
I20260926 19:30:14.789073 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:35185 every 8 connection(s)
I20260926 19:30:14.790702 14890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:14.794588 14957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:14.799913 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14890
I20260926 19:30:14.800163 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:30:14.800793 1180 external_mini_cluster.cc:1428] Setting key b871c2b8b4da7ce0ff08bed2d3aa2d16
I20260926 19:30:14.805886 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c: Bootstrap starting.
I20260926 19:30:14.809832 14957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:14.811147 14957 log.cc:824] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:14.814968 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c: No bootstrap required, opened a new log
Sep 26 19:30:14 dist-test-slave-9wb0 krb5kdc[14877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:14.824056 14957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } }
I20260926 19:30:14.824458 14957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:14.824613 14957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4e195d34384cb68be12716ccf3650c, State: Initialized, Role: FOLLOWER
I20260926 19:30:14.825227 14957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } }
I20260926 19:30:14.825562 14957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:14.825721 14957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:14.825911 14957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:14.826201 14960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:14.802674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.1.39.62:35185)
0926 19:30:14.803595 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:14.803608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:14.803652 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:30:14.805460 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:14.805489 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:14.805892 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:14.806272 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:14.808239 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:14.809574 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:14.811539 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:14.812722 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:14.812861 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:14.818366 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:14.818375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:14.818398 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:14.818450 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:14.821674 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:14.822444 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:14.822450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:14.822457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:14.822623 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:14.823029 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:14.823036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:14.823043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:14.823608 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:14.823912 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:14.824386 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 19:30:14.824883 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":466,"threads_started":1}
I20260926 19:30:14.828347 14957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } }
I20260926 19:30:14.828851 14957 leader_election.cc:302] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d4e195d34384cb68be12716ccf3650c; no voters:
I20260926 19:30:14.829559 14957 leader_election.cc:288] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:30:14.829704 14962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:14.830367 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:30:14.832110 14957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:14.832767 14962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [term 1 LEADER]: Becoming Leader. State: Replica: 1d4e195d34384cb68be12716ccf3650c, State: Running, Role: LEADER
I20260926 19:30:14.833493 14962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } }
I20260926 19:30:14.836854 14964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d4e195d34384cb68be12716ccf3650c. Latest consensus state: current_term: 1 leader_uuid: "1d4e195d34384cb68be12716ccf3650c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } } }
I20260926 19:30:14.836970 14963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d4e195d34384cb68be12716ccf3650c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d4e195d34384cb68be12716ccf3650c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 35185 } } }
I20260926 19:30:14.837252 14964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:14.837347 14963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:14.839534 14972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:14.846119 14972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:14.854988 14972 catalog_manager.cc:1347] Generated new cluster ID: b6d501b2a8ef4a24ba38257146f310ad
I20260926 19:30:14.855104 14972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:14.864297 14972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:30:14.865784 14972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:14.884807 14972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d4e195d34384cb68be12716ccf3650c: Generated new TSK 0
I20260926 19:30:14.885970 14972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:30:14.942592 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14890
I20260926 19:30:14.943789 14952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:15.041914 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14890
2026-09-26T19:30:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2039 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:30:15 dist-test-slave-9wb0 krb5kdc[14985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:30:15 dist-test-slave-9wb0 krb5kdc[14985](info): set up 2 sockets
Sep 26 19:30:15 dist-test-slave-9wb0 krb5kdc[14985](info): commencing operation
krb5kdc: starting...
W20260926 19:30:16.157858 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:30:16 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:30:16.413873 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:16.758778 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:16.759065 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:16.759155 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:16.766441 14998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:16.766558 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:16.766604 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:16.766669 14998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:16.766713 14998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:16.776640 14998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:44615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:16.777985 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:16.779295 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:16.791384 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:16.791458 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:16.791430 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:16.792130 14998 server_base.cc:1034] running on GCE node
I20260926 19:30:16.792918 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:16.794435 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:16.795688 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790451016795633 us; error 65 us; skew 500 ppm
I20260926 19:30:16.796043 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:16 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:16.799268 14998 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:16.800886 14998 webserver.cc:460] Webserver started at http://127.1.39.62:42871/ using document root <none> and password file <none>
I20260926 19:30:16.801694 14998 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:16.801849 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:16.802167 14998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:16.805233 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "cc2eca70614c4b4084298052bedfae36"
format_stamp: "Formatted at 2026-09-26 19:30:16 on dist-test-slave-9wb0"
server_key: "24c69d2997f387d99b14e2b71407447c"
server_key_iv: "0f3ed9639836792edaefa72d0c2c065d"
server_key_version: "encryptionkey@0"
I20260926 19:30:16.806068 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "cc2eca70614c4b4084298052bedfae36"
format_stamp: "Formatted at 2026-09-26 19:30:16 on dist-test-slave-9wb0"
server_key: "24c69d2997f387d99b14e2b71407447c"
server_key_iv: "0f3ed9639836792edaefa72d0c2c065d"
server_key_version: "encryptionkey@0"
I20260926 19:30:16.811278 14998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:30:16.814893 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:16.816273 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:30:16.816463 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cc2eca70614c4b4084298052bedfae36"
format_stamp: "Formatted at 2026-09-26 19:30:16 on dist-test-slave-9wb0"
server_key: "24c69d2997f387d99b14e2b71407447c"
server_key_iv: "0f3ed9639836792edaefa72d0c2c065d"
server_key_version: "encryptionkey@0"
I20260926 19:30:16.816680 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:16.835497 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:16.839706 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:16.840014 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:16.858692 14998 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:44615
I20260926 19:30:16.858737 15064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:44615 every 8 connection(s)
I20260926 19:30:16.860095 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:16.860734 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 14998
I20260926 19:30:16.860982 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:30:16.861759 1180 external_mini_cluster.cc:1428] Setting key 0eecb703bdd9adf3b13ec89d3e2d6e56
I20260926 19:30:16.864573 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:30:16 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:16.877701 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap starting.
I20260926 19:30:16.881893 15065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:16.883215 15065 log.cc:824] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:16.886580 15065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: No bootstrap required, opened a new log
I20260926 19:30:16.886765 15068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:16.863821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.1.39.62:44615)
0926 19:30:16.864649 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:16.864666 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:16.864715 (+ 49us) server_negotiation.cc:407] Connection header received
0926 19:30:16.867050 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:16.867079 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:16.867614 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:16.868065 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:16.869393 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:16.871473 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:16.872795 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:16.873842 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:16.874037 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:16.878156 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:16.878167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:16.878185 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:16.878246 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:16.882326 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:16.883017 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:16.883026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:16.883034 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:16.883261 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:16.883680 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:16.883688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:16.883694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:16.884315 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:16.884625 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:16.885025 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 19:30:16.885477 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":348,"threads_started":1}
I20260926 19:30:16.895372 15065 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:16.895725 15065 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:16.895799 15065 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Initialized, Role: FOLLOWER
I20260926 19:30:16.896270 15065 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:16.896489 15065 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:16.896580 15065 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:16.896703 15065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:16.899143 15065 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:16.899695 15065 leader_election.cc:302] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2eca70614c4b4084298052bedfae36; no voters:
I20260926 19:30:16.900282 15065 leader_election.cc:288] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:30:16.900434 15070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:16.901147 15070 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 LEADER]: Becoming Leader. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Running, Role: LEADER
I20260926 19:30:16.901769 15070 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:16.902420 15065 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:16.904417 15071 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:16.904770 15071 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:16.910048 15072 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2eca70614c4b4084298052bedfae36. Latest consensus state: current_term: 1 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:16.911178 15072 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:16.911830 15075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:16.916680 15075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:16.926662 15075 catalog_manager.cc:1347] Generated new cluster ID: 21ffe4829dc349dab09dd414decd12bb
I20260926 19:30:16.926796 15075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:16.952031 15075 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:30:16.954064 15075 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:16.963271 15075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Generated new TSK 0
I20260926 19:30:16.964398 15075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:30:17.022928 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:17.362323 15091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:17.362643 15091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:17.362808 15091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:17.370033 15091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:17.370205 15091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:17.370378 15091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:17.380254 15091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:17.381762 15091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:17.383181 15091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:17.398192 15096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:17.398236 15099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:17.398161 15097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:17.399681 15091 server_base.cc:1034] running on GCE node
I20260926 19:30:17.400358 15091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:17.401522 15091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:17.402760 15091 hybrid_clock.cc:648] HybridClock initialized: now 1790451017402685 us; error 67 us; skew 500 ppm
I20260926 19:30:17.403157 15091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:17 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:17.406490 15091 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:17.408034 15091 webserver.cc:460] Webserver started at http://127.1.39.1:42703/ using document root <none> and password file <none>
I20260926 19:30:17.408794 15091 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:17.408946 15091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:17.409271 15091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:17.412485 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "60fd5e667ac445c0a771cabc369e0852"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "bcdcf94d05758d9a07ec135a971bc258"
server_key_iv: "a1bbc85ef97fd6b9df2e44f958095d5f"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.413367 15091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "60fd5e667ac445c0a771cabc369e0852"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "bcdcf94d05758d9a07ec135a971bc258"
server_key_iv: "a1bbc85ef97fd6b9df2e44f958095d5f"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.418771 15091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:30:17.422617 15106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:17.423945 15091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:30:17.424135 15091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "60fd5e667ac445c0a771cabc369e0852"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "bcdcf94d05758d9a07ec135a971bc258"
server_key_iv: "a1bbc85ef97fd6b9df2e44f958095d5f"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.424360 15091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:17.451992 15091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:17.456461 15091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:17.456776 15091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:17.457814 15091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:17.459213 15091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:17.459292 15091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:17.459420 15091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:17.459483 15091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:17.493806 15091 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:35227
I20260926 19:30:17.493855 15218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:35227 every 8 connection(s)
I20260926 19:30:17.495075 15091 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:17.499962 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15091
I20260926 19:30:17.500188 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:30:17.501010 1180 external_mini_cluster.cc:1428] Setting key 96f6d3672f5fa7b02dc63970bd31e872
Sep 26 19:30:17 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:17.516068 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:17.497784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:45293 (local address 127.1.39.62:44615)
0926 19:30:17.498262 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:17.498266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:17.498919 (+ 653us) server_negotiation.cc:407] Connection header received
0926 19:30:17.501513 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:17.501517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:17.501586 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:17.501715 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:17.503393 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:17.504209 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:17.505864 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:17.506253 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:17.506399 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:17.510081 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:17.510086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:17.510089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:17.510122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:17.512916 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:17.513767 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:17.513772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:17.513774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:17.513897 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:17.514335 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:17.514346 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:17.514349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:17.514607 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:17.514763 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:17.515475 (+ 712us) server_negotiation.cc:299] Negotiation successful
0926 19:30:17.515706 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260926 19:30:17.517158 15222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:17.498083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:45293)
0926 19:30:17.498758 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:17.498804 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:17.501168 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:17.501959 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:17.501968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:17.502536 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:17.503193 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:17.503238 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:17.504375 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:17.504380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:17.505703 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:17.505715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:17.506486 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:17.506494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:17.506735 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:17.507450 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:17.507473 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:17.509853 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:17.513052 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:17.513061 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:17.513078 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:17.513633 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:17.514006 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:17.514012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:17.514015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:17.514212 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:17.514737 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:17.514745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:17.515079 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 19:30:17.515353 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 19:30:17.515869 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":267,"threads_started":1}
I20260926 19:30:17.518911 15219 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:17.519228 15219 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:17.520076 15219 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:17.522893 15030 ts_manager.cc:194] Registered new tserver with Master: 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1:35227)
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:30:17.525197 15030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:45293
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:30:17.557992 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:17.895582 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:17.895877 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:17.895972 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:17.903450 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:17.903579 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:17.903697 15225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:17.915282 15225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:17.916571 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:17.917929 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:17.931300 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:17.931344 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:17.931325 15233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:17.932406 15225 server_base.cc:1034] running on GCE node
I20260926 19:30:17.933038 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:17.934023 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:17.935235 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790451017935189 us; error 32 us; skew 500 ppm
I20260926 19:30:17.935566 15225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:17 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:17.938733 15225 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:17.940295 15225 webserver.cc:460] Webserver started at http://127.1.39.2:40551/ using document root <none> and password file <none>
I20260926 19:30:17.941008 15225 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:17.941162 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:17.941530 15225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:17.944492 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "abf8d703f4be4616ae26223644903e41"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "5be17414411667a59e9e8447318418d8"
server_key_iv: "b9d3f37a8b8ec2dffc0b38d938119514"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.945272 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "abf8d703f4be4616ae26223644903e41"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "5be17414411667a59e9e8447318418d8"
server_key_iv: "b9d3f37a8b8ec2dffc0b38d938119514"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.951622 15225 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 19:30:17.955495 15240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:17.956936 15225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:30:17.957129 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "abf8d703f4be4616ae26223644903e41"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "5be17414411667a59e9e8447318418d8"
server_key_iv: "b9d3f37a8b8ec2dffc0b38d938119514"
server_key_version: "encryptionkey@0"
I20260926 19:30:17.957365 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:17.968912 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:17.973754 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:17.974073 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:17.975095 15225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:17.976496 15225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:17.976610 15225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:17.976756 15225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:17.976830 15225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:18.008531 15225 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:42133
I20260926 19:30:18.008569 15352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:42133 every 8 connection(s)
I20260926 19:30:18.009967 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:18.013347 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15225
I20260926 19:30:18.013586 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:30:18.014235 1180 external_mini_cluster.cc:1428] Setting key 71cb5e3e6b3c4d8fb4b4ae6d1bae32f2
Sep 26 19:30:18 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451017, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:18.035996 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.012830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:34033 (local address 127.1.39.62:44615)
0926 19:30:18.013077 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.013084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.014580 (+ 1496us) server_negotiation.cc:407] Connection header received
0926 19:30:18.016867 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.016872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.016950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.017036 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:18.019336 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.020145 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.022806 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.023248 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.023354 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.027841 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:18.027851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:18.027859 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:18.027907 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:18.032034 (+ 4127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.032924 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.032932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.032939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.033172 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.033791 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.033799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.033805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.034277 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:18.034432 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.035257 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.035535 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 19:30:18.036669 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.013432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:34033)
0926 19:30:18.014364 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.014424 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.016540 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.017368 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.017384 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.018235 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:18.019116 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.019144 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.020316 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.020324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.022642 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.022666 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.023490 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.023500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.023845 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.024592 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:18.024635 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:18.027568 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:18.032218 (+ 4650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:18.032226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:18.032241 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:18.032755 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:18.033342 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:18.033350 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:18.033355 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:18.033610 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:18.034431 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:18.034442 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:18.034822 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.035062 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.035496 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":362,"threads_started":1}
I20260926 19:30:18.038393 15353 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:18.038731 15353 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:18.039450 15353 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:18.041010 15030 ts_manager.cc:194] Registered new tserver with Master: abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133)
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
I20260926 19:30:18.041899 15030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:34033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:30:18.073668 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:18.413969 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:18.414469 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:18.414685 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:18.427431 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:18.427635 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:18.427881 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:18.443787 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:18.445705 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:18.447714 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:18.467043 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:18.467052 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:18.467131 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:18.473430 15358 server_base.cc:1034] running on GCE node
I20260926 19:30:18.474202 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:18.475195 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:18.476433 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790451018476381 us; error 42 us; skew 500 ppm
I20260926 19:30:18.476853 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:18 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:18.480212 15358 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:18.482128 15358 webserver.cc:460] Webserver started at http://127.1.39.3:33691/ using document root <none> and password file <none>
I20260926 19:30:18.482896 15358 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:18.483045 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:18.483428 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:18.489952 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
format_stamp: "Formatted at 2026-09-26 19:30:18 on dist-test-slave-9wb0"
server_key: "ffdbe1bd8642b157d5303a5c31844d14"
server_key_iv: "b57b906923b618cf0eddc70ba1702489"
server_key_version: "encryptionkey@0"
I20260926 19:30:18.492130 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
format_stamp: "Formatted at 2026-09-26 19:30:18 on dist-test-slave-9wb0"
server_key: "ffdbe1bd8642b157d5303a5c31844d14"
server_key_iv: "b57b906923b618cf0eddc70ba1702489"
server_key_version: "encryptionkey@0"
I20260926 19:30:18.499868 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 19:30:18.503485 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:18.504842 15358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:30:18.504978 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
format_stamp: "Formatted at 2026-09-26 19:30:18 on dist-test-slave-9wb0"
server_key: "ffdbe1bd8642b157d5303a5c31844d14"
server_key_iv: "b57b906923b618cf0eddc70ba1702489"
server_key_version: "encryptionkey@0"
I20260926 19:30:18.505160 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:18.527185 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:18.528832 15219 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:18.531951 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:18.532194 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:18.533107 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:18.534436 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:18.534512 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:18.534593 15358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:18.534646 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:18.569000 15358 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:45811
I20260926 19:30:18.569108 15485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:45811 every 8 connection(s)
I20260926 19:30:18.570362 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:18.578922 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15358
I20260926 19:30:18.579175 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:30:18.579744 1180 external_mini_cluster.cc:1428] Setting key d5f1cb97ac689b7dff1a10761bae673e
Sep 26 19:30:18 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451018, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:18.591058 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.573230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:32887 (local address 127.1.39.62:44615)
0926 19:30:18.573775 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.573780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.574556 (+ 776us) server_negotiation.cc:407] Connection header received
0926 19:30:18.576430 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.576436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.576526 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.576653 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:18.578240 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.579032 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.581191 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.581674 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.581840 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.585436 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:18.585441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:18.585444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:18.585478 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:18.587889 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.588623 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.588634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.588639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.588770 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.589197 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.589207 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.589209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.589528 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:18.589685 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.590418 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.590674 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
I20260926 19:30:18.591951 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.573666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:32887)
0926 19:30:18.574393 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.574429 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.576158 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.576886 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.576895 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.577494 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:18.578056 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.578072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.579221 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.579227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.581041 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.581053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.581922 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.581931 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.582176 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.582889 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:18.582910 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:18.585179 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:18.588055 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:18.588063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:18.588088 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:18.588491 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:18.588882 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:18.588887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:18.588891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:18.589072 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:18.589663 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:18.589670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:18.589995 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.590273 (+ 278us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.590832 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":273,"threads_started":1}
I20260926 19:30:18.593431 15486 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:18.593768 15486 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:18.594590 15486 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:18.596191 15030 ts_manager.cc:194] Registered new tserver with Master: d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
I20260926 19:30:18.596930 15030 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:32887
I20260926 19:30:18.607537 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:30:18.624794 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.612066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.1.39.62:44615)
0926 19:30:18.612316 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.612325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.612747 (+ 422us) server_negotiation.cc:407] Connection header received
0926 19:30:18.612899 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.612903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.612967 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.613114 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:18.614528 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.615401 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.616707 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.617084 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.617174 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.620017 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:18.620022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:18.620025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:18.620057 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:18.622613 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.623223 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.623227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.623229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.623343 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:18.623660 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:18.623663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:18.623665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:18.623983 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:18.624160 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.624396 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.624563 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 19:30:18.629189 15030 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:30:18.635998 15030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:30:18.659287 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.647746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:45811 (local address 127.0.0.1:38098)
0926 19:30:18.648736 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.648785 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.648958 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.652508 (+ 3550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.652512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.652536 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:18.653005 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.653019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.654905 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.654911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.656972 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.656990 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.658404 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.658410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.658608 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.658664 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.658766 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.658954 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":127,"thread_start_us":565,"threads_started":1}
I20260926 19:30:18.659834 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.648080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:40136)
0926 19:30:18.649532 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.649553 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.649662 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.651961 (+ 2299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.651965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.651993 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:18.652407 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.652427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.654037 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.654044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.657076 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.657084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.658997 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.659004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.659182 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.659213 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.659292 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.659425 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":234,"thread_start_us":1145,"threads_started":1}
I20260926 19:30:18.660351 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.648127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.1.39.3:45811)
0926 19:30:18.651734 (+ 3607us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.651742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.651777 (+ 35us) server_negotiation.cc:407] Connection header received
0926 19:30:18.651864 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.651871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.652300 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.653193 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:18.653505 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.654772 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.657133 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.658190 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.659545 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.659786 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.659839 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.660016 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3442,"thread_start_us":3284,"threads_started":1}
I20260926 19:30:18.661020 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.649662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42133 (local address 127.0.0.1:56030)
0926 19:30:18.650227 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.650249 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.650376 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.651635 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.651643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.651674 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:18.652064 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.652077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.654308 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.654314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.657041 (+ 2727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.657061 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.660144 (+ 3083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.660153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.660331 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.660354 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.660461 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.660609 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":234,"thread_start_us":836,"threads_started":1}
I20260926 19:30:18.661770 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.648724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40136 (local address 127.1.39.1:35227)
0926 19:30:18.651462 (+ 2738us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.651470 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.651497 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:30:18.651589 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.651597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.651774 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.651958 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:18.652638 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.653900 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.657244 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.658744 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.660915 (+ 2171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.661106 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.661185 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.661398 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2534,"thread_start_us":2348,"threads_started":1}
I20260926 19:30:18.662055 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.650287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.1.39.2:42133)
0926 19:30:18.650969 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.650981 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.651014 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:30:18.651120 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.651127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.651380 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.652274 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:18.652513 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.654171 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.657244 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.658744 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.658997 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.659205 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.661546 (+ 2341us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.661730 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":365,"threads_started":1}
I20260926 19:30:18.664685 15154 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (DEFAULT_TABLE table=test-table [id=9c0a3a329e774a1db2b9c00743787c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:18.664620 15421 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (DEFAULT_TABLE table=test-table [id=9c0a3a329e774a1db2b9c00743787c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:18.665184 15288 tablet_service.cc:1467] Processing CreateTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (DEFAULT_TABLE table=test-table [id=9c0a3a329e774a1db2b9c00743787c36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:18.666169 15154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91a08ca32e4fb993fc9fd1258b91a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:18.666430 15288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91a08ca32e4fb993fc9fd1258b91a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:18.666427 15421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f91a08ca32e4fb993fc9fd1258b91a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:18.676990 15505 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Bootstrap starting.
I20260926 19:30:18.681355 15505 tablet_bootstrap.cc:654] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:18.681388 15506 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Bootstrap starting.
I20260926 19:30:18.681746 15507 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Bootstrap starting.
I20260926 19:30:18.683355 15505 log.cc:824] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:18.684803 15506 tablet_bootstrap.cc:654] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:18.684924 15507 tablet_bootstrap.cc:654] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:18.685930 15507 log.cc:824] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:18.686053 15506 log.cc:824] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:18.687018 15505 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: No bootstrap required, opened a new log
I20260926 19:30:18.687309 15505 ts_tablet_manager.cc:1397] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260926 19:30:18.689105 15507 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: No bootstrap required, opened a new log
I20260926 19:30:18.689411 15506 tablet_bootstrap.cc:492] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: No bootstrap required, opened a new log
I20260926 19:30:18.689399 15507 ts_tablet_manager.cc:1397] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:30:18.689644 15506 ts_tablet_manager.cc:1397] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:30:18.694686 15505 raft_consensus.cc:348] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.695073 15505 raft_consensus.cc:374] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:18.695166 15505 raft_consensus.cc:729] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fd5e667ac445c0a771cabc369e0852, State: Initialized, Role: FOLLOWER
I20260926 19:30:18.695815 15505 consensus_queue.cc:260] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.698731 15505 ts_tablet_manager.cc:1428] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260926 19:30:18.700213 15506 raft_consensus.cc:348] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.700213 15507 raft_consensus.cc:348] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.700646 15507 raft_consensus.cc:374] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:18.700646 15506 raft_consensus.cc:374] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:18.700757 15507 raft_consensus.cc:729] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d100a3bced8e4fbc8b0aa0a151afe0c3, State: Initialized, Role: FOLLOWER
I20260926 19:30:18.700757 15506 raft_consensus.cc:729] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf8d703f4be4616ae26223644903e41, State: Initialized, Role: FOLLOWER
I20260926 19:30:18.701364 15506 consensus_queue.cc:260] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.701362 15507 consensus_queue.cc:260] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.703125 15486 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:18.703091 15353 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:18.703171 15507 ts_tablet_manager.cc:1428] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260926 19:30:18.703567 15506 ts_tablet_manager.cc:1428] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260926 19:30:18.747618 15220 tablet.cc:2369] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:18.762126 15354 tablet.cc:2369] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:18.822063 15487 tablet.cc:2369] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:18.867858 15511 raft_consensus.cc:482] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:18.868185 15511 raft_consensus.cc:504] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.870337 15511 leader_election.cc:288] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811), abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133)
I20260926 19:30:18.877380 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.870773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:55285 (local address 127.1.39.2:42133)
0926 19:30:18.871007 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.871013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.871885 (+ 872us) server_negotiation.cc:407] Connection header received
0926 19:30:18.872328 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.872334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.872409 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.872507 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:18.873069 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.873887 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.875658 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.876416 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.876568 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.876676 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.876960 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.877093 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:30:18.877509 15514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.870745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42133 (local address 127.1.39.1:55285)
0926 19:30:18.871747 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.871774 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.872022 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.872565 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.872570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.872596 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:18.872921 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.872929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.874013 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.874020 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.875484 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.875496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.876655 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.876660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.876802 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.876825 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.876933 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.877075 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":1,"thread_start_us":637,"threads_started":1}
I20260926 19:30:18.877815 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.870857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:60931 (local address 127.1.39.3:45811)
0926 19:30:18.871092 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.871100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.872319 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 19:30:18.872554 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.872559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.872628 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.872696 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:18.873582 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.874317 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.876063 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.876823 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.876968 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.877077 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.877408 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.877557 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:30:18.878211 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.871287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:45811 (local address 127.1.39.1:60931)
0926 19:30:18.872218 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:18.872236 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:18.872415 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:18.872884 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:18.872889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:18.872915 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:18.873342 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.873358 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.874463 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.874470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.875897 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:18.875908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.877056 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.877064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:18.877215 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.877237 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 19:30:18.877380 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 19:30:18.877801 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":45,"thread_start_us":535,"threads_started":1}
I20260926 19:30:18.878700 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f91a08ca32e4fb993fc9fd1258b91a1" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf8d703f4be4616ae26223644903e41" is_pre_election: true
I20260926 19:30:18.879252 15308 raft_consensus.cc:2454] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 0.
I20260926 19:30:18.879248 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f91a08ca32e4fb993fc9fd1258b91a1" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" is_pre_election: true
I20260926 19:30:18.879807 15441 raft_consensus.cc:2454] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 0.
I20260926 19:30:18.879961 15110 leader_election.cc:302] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852, abf8d703f4be4616ae26223644903e41; no voters:
I20260926 19:30:18.880380 15511 raft_consensus.cc:2781] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:30:18.880616 15511 raft_consensus.cc:482] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:30:18.880735 15511 raft_consensus.cc:3037] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:18.883534 15511 raft_consensus.cc:504] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.884279 15511 leader_election.cc:288] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 election: Requested vote from peers d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811), abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133)
I20260926 19:30:18.885139 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f91a08ca32e4fb993fc9fd1258b91a1" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
I20260926 19:30:18.885159 15308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f91a08ca32e4fb993fc9fd1258b91a1" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf8d703f4be4616ae26223644903e41"
I20260926 19:30:18.885390 15308 raft_consensus.cc:3037] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:18.885401 15441 raft_consensus.cc:3037] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:18.888571 15308 raft_consensus.cc:2454] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 1.
I20260926 19:30:18.889032 15110 leader_election.cc:302] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852, abf8d703f4be4616ae26223644903e41; no voters:
I20260926 19:30:18.889091 15441 raft_consensus.cc:2454] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 1.
I20260926 19:30:18.889426 15511 raft_consensus.cc:2781] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:18.889979 15511 raft_consensus.cc:686] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 1 LEADER]: Becoming Leader. State: Replica: 60fd5e667ac445c0a771cabc369e0852, State: Running, Role: LEADER
I20260926 19:30:18.890625 15511 consensus_queue.cc:237] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } }
I20260926 19:30:18.895816 15029 catalog_manager.cc:5495] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "60fd5e667ac445c0a771cabc369e0852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } health_report { overall_health: UNKNOWN } } }
I20260926 19:30:18.955502 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.950006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.1.39.1:35227)
0926 19:30:18.950238 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.950243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:18.950261 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:30:18.950411 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:18.950414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:18.950501 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:18.950629 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:30:18.951021 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.951766 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.953218 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:18.953807 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:18.953901 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:18.954363 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:30:18.954867 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:30:18.954973 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:18.955125 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 19:30:18.955196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 19:30:18.963605 15308 raft_consensus.cc:1261] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Refusing update from remote peer 60fd5e667ac445c0a771cabc369e0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:18.963639 15441 raft_consensus.cc:1261] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Refusing update from remote peer 60fd5e667ac445c0a771cabc369e0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:18.964592 15516 consensus_queue.cc:1035] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:18.964934 15511 consensus_queue.cc:1035] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:18.988871 15030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:18.989177 15030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:18.992677 15030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Sending DeleteTablet for 3 replicas of tablet 4f91a08ca32e4fb993fc9fd1258b91a1
I20260926 19:30:18.993829 15421 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:18 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:38098
I20260926 19:30:18.994037 15154 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:18 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:40136
I20260926 19:30:18.994067 15288 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f91a08ca32e4fb993fc9fd1258b91a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:18 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:56030
I20260926 19:30:18.995939 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15091
I20260926 19:30:19.011220 15529 tablet_replica.cc:330] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: stopping tablet replica
I20260926 19:30:19.011754 15529 raft_consensus.cc:2229] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:30:19.012552 15529 raft_consensus.cc:2258] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:19.014869 15529 ts_tablet_manager.cc:1905] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:19.011297 15528 tablet_replica.cc:330] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: stopping tablet replica
I20260926 19:30:19.015506 15528 raft_consensus.cc:2229] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:19.016088 15528 raft_consensus.cc:2258] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:19.017125 15530 tablet_replica.cc:330] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: stopping tablet replica
I20260926 19:30:19.017554 15530 raft_consensus.cc:2229] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:19.018042 15530 raft_consensus.cc:2258] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:19.018359 15528 ts_tablet_manager.cc:1905] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:19.022341 15530 ts_tablet_manager.cc:1905] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:30:19 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:30:19.029986 15530 ts_tablet_manager.cc:1918] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:19.030200 15530 log.cc:1196] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/4f91a08ca32e4fb993fc9fd1258b91a1
I20260926 19:30:19.030783 15530 ts_tablet_manager.cc:1939] T 4f91a08ca32e4fb993fc9fd1258b91a1 P abf8d703f4be4616ae26223644903e41: Deleting consensus metadata
I20260926 19:30:19.032886 15017 catalog_manager.cc:4841] TS abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (table test-table [id=9c0a3a329e774a1db2b9c00743787c36]) successfully deleted
I20260926 19:30:19.037350 15529 ts_tablet_manager.cc:1918] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:19.037554 15529 log.cc:1196] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/4f91a08ca32e4fb993fc9fd1258b91a1
I20260926 19:30:19.038136 15529 ts_tablet_manager.cc:1939] T 4f91a08ca32e4fb993fc9fd1258b91a1 P 60fd5e667ac445c0a771cabc369e0852: Deleting consensus metadata
I20260926 19:30:19.039901 15502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:18.996791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.1.39.1:35227)
0926 19:30:18.997009 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:18.997019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:19.004658 (+ 7639us) server_negotiation.cc:407] Connection header received
0926 19:30:19.005785 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:19.005795 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:19.005912 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:19.006003 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:19.011593 (+ 5590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.012777 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.019769 (+ 6992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.020490 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.020617 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:19.029424 (+ 8807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:19.029440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:19.029459 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:19.029517 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:19.034009 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.036795 (+ 2786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.036804 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.036808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.037017 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.037804 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.037813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.037819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.038377 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:19.038666 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:19.039263 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 19:30:19.039524 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 19:30:19.040084 15017 catalog_manager.cc:4841] TS 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1:35227): tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (table test-table [id=9c0a3a329e774a1db2b9c00743787c36]) successfully deleted
I20260926 19:30:19.040815 15528 ts_tablet_manager.cc:1918] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:19.041002 15528 log.cc:1196] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/4f91a08ca32e4fb993fc9fd1258b91a1
I20260926 19:30:19.041606 15528 ts_tablet_manager.cc:1939] T 4f91a08ca32e4fb993fc9fd1258b91a1 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting consensus metadata
I20260926 19:30:19.041713 15214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:19.043457 15015 catalog_manager.cc:4841] TS d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): tablet 4f91a08ca32e4fb993fc9fd1258b91a1 (table test-table [id=9c0a3a329e774a1db2b9c00743787c36]) successfully deleted
I20260926 19:30:19.144301 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15091
I20260926 19:30:19.164013 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15225
Sep 26 19:30:19 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:30:19.179551 15504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:19.164811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.1.39.2:42133)
0926 19:30:19.164996 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:19.165004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:19.165025 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:30:19.165238 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:19.165244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:19.165372 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:19.165476 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:19.166764 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.167540 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.168904 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.169365 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.169495 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:19.173913 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:19.173929 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:19.173946 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:19.173984 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:19.176689 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.177435 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.177447 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.177455 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.177584 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.177955 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.177961 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.177964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.178354 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:19.178664 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:19.178981 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 19:30:19.179220 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:30:19.180274 15348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:19.270792 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15225
I20260926 19:30:19.288744 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15358
Sep 26 19:30:19 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:30:19.305514 15501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:19.289679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.1.39.3:45811)
0926 19:30:19.289867 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:19.289872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:19.289992 (+ 120us) server_negotiation.cc:407] Connection header received
0926 19:30:19.290163 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:19.290167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:19.290246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:19.290336 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:19.291725 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.292408 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.294188 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:19.294623 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:19.294722 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:19.299665 (+ 4943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:19.299684 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:19.299701 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:19.299749 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:19.302787 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.303553 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.303561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.303563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.303713 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:19.304037 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:19.304040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:19.304042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:19.304414 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:19.304641 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:19.304930 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 19:30:19.305123 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:30:19.306209 15481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:19.399219 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15358
I20260926 19:30:19.418897 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 14998
I20260926 19:30:19.420127 15060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:19.533608 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 14998
I20260926 19:30:19.556304 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--webserver_interface=127.1.39.62
--webserver_port=42871
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:19.895821 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:19.896154 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:19.896324 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:19.903725 15536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:19.903846 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:19.903955 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:19.904036 15536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:19.904114 15536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:19.914296 15536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:44615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=42871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:19.915642 15536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:19.917189 15536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:19.942379 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:19.942368 15545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:19.942463 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:19.947265 15536 server_base.cc:1034] running on GCE node
I20260926 19:30:19.948280 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:19.950160 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:19.951433 15536 hybrid_clock.cc:648] HybridClock initialized: now 1790451019951391 us; error 62 us; skew 500 ppm
I20260926 19:30:19.951858 15536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:19 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451019, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:19.960160 15536 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:19.962209 15536 webserver.cc:460] Webserver started at http://127.1.39.62:42871/ using document root <none> and password file <none>
I20260926 19:30:19.963022 15536 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:19.963201 15536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:19.969436 15536 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:30:19.972450 15552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:19.973923 15536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:19.974114 15536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cc2eca70614c4b4084298052bedfae36"
format_stamp: "Formatted at 2026-09-26 19:30:16 on dist-test-slave-9wb0"
server_key: "24c69d2997f387d99b14e2b71407447c"
server_key_iv: "0f3ed9639836792edaefa72d0c2c065d"
server_key_version: "encryptionkey@0"
I20260926 19:30:19.975209 15536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:19.988088 15536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:19.992384 15536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:19.992678 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:20.012562 15536 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:44615
I20260926 19:30:20.012593 15603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:44615 every 8 connection(s)
I20260926 19:30:20.014300 15536 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:20.021349 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15536
I20260926 19:30:20.021992 15604 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:30:20.022455 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:35227
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=42703
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:20.027673 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap starting.
I20260926 19:30:20.049126 15604 log.cc:824] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:20.061656 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:30:20.062258 15604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap complete.
I20260926 19:30:20.071731 15604 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:20.072692 15604 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Initialized, Role: FOLLOWER
I20260926 19:30:20.073463 15604 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:20.073711 15604 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:20.073829 15604 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:20.074028 15604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:30:20.078034 15604 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:20.078629 15604 leader_election.cc:302] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2eca70614c4b4084298052bedfae36; no voters:
I20260926 19:30:20.079331 15604 leader_election.cc:288] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:30:20.079479 15608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:30:20.080839 15608 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 LEADER]: Becoming Leader. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Running, Role: LEADER
I20260926 19:30:20.081451 15608 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:20.081777 15604 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:20.084260 15609 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:20.084677 15609 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:20.086406 15610 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2eca70614c4b4084298052bedfae36. Latest consensus state: current_term: 2 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:20.086905 15610 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:20.087317 15613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:20.095587 15613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:20.097062 15613 catalog_manager.cc:1259] Loaded cluster ID: 21ffe4829dc349dab09dd414decd12bb
I20260926 19:30:20.097234 15613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:20.100529 15613 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 19:30:20.118706 15613 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 19:30:20.144083 15613 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:20.146318 15613 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Loaded TSK: 0
I20260926 19:30:20.148275 15613 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:30:20.396175 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:20.396530 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:20.396697 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:20.404142 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:20.404315 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:20.404489 15606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:20.414894 15606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=42703
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:20.416289 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:20.417734 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:20.432775 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:20.433175 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:20.433697 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:20.434572 15606 server_base.cc:1034] running on GCE node
I20260926 19:30:20.435504 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:20.436543 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:20.437783 15606 hybrid_clock.cc:648] HybridClock initialized: now 1790451020437720 us; error 59 us; skew 500 ppm
I20260926 19:30:20.438127 15606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:20 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:20.441711 15606 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:20.443583 15606 webserver.cc:460] Webserver started at http://127.1.39.1:42703/ using document root <none> and password file <none>
I20260926 19:30:20.444345 15606 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:20.444486 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:20.450333 15606 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 19:30:20.453399 15641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.454962 15606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:30:20.455171 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "60fd5e667ac445c0a771cabc369e0852"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "bcdcf94d05758d9a07ec135a971bc258"
server_key_iv: "a1bbc85ef97fd6b9df2e44f958095d5f"
server_key_version: "encryptionkey@0"
I20260926 19:30:20.456254 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:20.476732 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:20.481833 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:20.482161 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:20.483359 15606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:20.489918 15606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:20.490048 15606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.490211 15606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:20.490319 15606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.524899 15606 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:35227
I20260926 19:30:20.524986 15753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:35227 every 8 connection(s)
I20260926 19:30:20.526417 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:20.530704 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15606
I20260926 19:30:20.531299 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:42133
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=40551
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:20 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:20.551610 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:20.529343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:34151)
0926 19:30:20.530178 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:20.530237 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:20.532070 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:20.533741 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:20.533748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:20.534274 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:20.534786 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:20.534803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.537339 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.537346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:20.538535 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:20.538546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.539964 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.539972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:20.540264 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:20.541071 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:20.541098 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:20.543590 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:20.547119 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:20.547129 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:20.547146 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:20.547572 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:20.548029 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:20.548036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:20.548040 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:20.548242 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:20.549016 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:20.549024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:20.549423 (+ 399us) client_negotiation.cc:769] Sending connection context
0926 19:30:20.549695 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 19:30:20.550285 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":340,"threads_started":1}
I20260926 19:30:20.551632 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:20.529193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:34151 (local address 127.1.39.62:44615)
0926 19:30:20.529849 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:20.529860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:20.530393 (+ 533us) server_negotiation.cc:407] Connection header received
0926 19:30:20.532706 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:20.532726 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:20.533189 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:20.533640 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:20.535022 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.537131 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.538673 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.539707 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.539922 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:20.543871 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:20.543879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:20.543892 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:20.543929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:20.546952 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:20.547731 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:20.547738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:20.547743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:20.547904 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:20.548374 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:20.548378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:20.548380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:20.548871 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:20.549080 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:20.549915 (+ 835us) server_negotiation.cc:299] Negotiation successful
0926 19:30:20.550316 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
I20260926 19:30:20.554901 15754 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:20.555222 15754 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:20.555958 15754 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:20.558254 15569 ts_manager.cc:194] Registered new tserver with Master: 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1:35227)
I20260926 19:30:20.560268 15569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:34151
W20260926 19:30:20.864634 15758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:20.864924 15758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:20.865092 15758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:20.872731 15758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:20.872861 15758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:20.873098 15758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:20.883857 15758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:42133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=40551
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:20.885360 15758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:20.886745 15758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:20.901634 15766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:20.901671 15764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:20.901716 15763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:20.903540 15758 server_base.cc:1034] running on GCE node
I20260926 19:30:20.904225 15758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:20.905412 15758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:20.906661 15758 hybrid_clock.cc:648] HybridClock initialized: now 1790451020906616 us; error 39 us; skew 500 ppm
I20260926 19:30:20.907017 15758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:20 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:20.910617 15758 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:20.912189 15758 webserver.cc:460] Webserver started at http://127.1.39.2:40551/ using document root <none> and password file <none>
I20260926 19:30:20.912953 15758 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:20.913116 15758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:20.918717 15758 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 19:30:20.921766 15773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.923460 15758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:30:20.923666 15758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "abf8d703f4be4616ae26223644903e41"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "5be17414411667a59e9e8447318418d8"
server_key_iv: "b9d3f37a8b8ec2dffc0b38d938119514"
server_key_version: "encryptionkey@0"
I20260926 19:30:20.924885 15758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:20.943321 15758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:20.947453 15758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:20.947753 15758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:20.948743 15758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:20.950152 15758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:20.950292 15758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.950439 15758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:20.950544 15758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:20.983664 15758 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:42133
I20260926 19:30:20.983753 15885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:42133 every 8 connection(s)
I20260926 19:30:20.985141 15758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:20.987514 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15758
I20260926 19:30:20.988263 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:45811
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=33691
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:30:20 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451020, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:21.006422 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:20.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:35335 (local address 127.1.39.62:44615)
0926 19:30:20.987929 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:20.987933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:20.989442 (+ 1509us) server_negotiation.cc:407] Connection header received
0926 19:30:20.991466 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:20.991472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:20.991569 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:20.991685 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:20.993197 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.993972 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.995646 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.996067 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.996183 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.000471 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:21.000477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:21.000480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:21.000514 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:21.003281 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.004060 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.004066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.004076 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.004204 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.004635 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.004640 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.004645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.004911 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:21.005064 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.005823 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.006060 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:30:21.007560 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:20.988419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:35335)
0926 19:30:20.989227 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:20.989271 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:20.991179 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:20.991930 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:20.991939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:20.992457 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:20.993010 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:20.993028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.994136 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.994149 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:20.995472 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:20.995490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:20.996311 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:20.996321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:20.996684 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:20.997496 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:20.997530 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:21.000233 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:21.003455 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:21.003465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:21.003484 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:21.003923 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:21.004319 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:21.004324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:21.004328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:21.004514 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:21.005048 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:21.005056 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:21.005464 (+ 408us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.005740 (+ 276us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.006293 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":541,"threads_started":1}
I20260926 19:30:21.009208 15886 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:21.009591 15886 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:21.010361 15886 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:21.012009 15569 ts_manager.cc:194] Registered new tserver with Master: abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133)
I20260926 19:30:21.013453 15569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:35335
W20260926 19:30:21.335738 15889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:21.336042 15889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:21.336160 15889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:21.344514 15889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:21.344645 15889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:21.344770 15889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:21.355348 15889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:45811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=33691
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:21.356607 15889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:21.357854 15889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:21.371057 15897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:21.371177 15895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:21.371177 15894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:21.371726 15889 server_base.cc:1034] running on GCE node
I20260926 19:30:21.372377 15889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:21.373437 15889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:21.374686 15889 hybrid_clock.cc:648] HybridClock initialized: now 1790451021374648 us; error 44 us; skew 500 ppm
I20260926 19:30:21.375021 15889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:21 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:21.378476 15889 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:21.380326 15889 webserver.cc:460] Webserver started at http://127.1.39.3:33691/ using document root <none> and password file <none>
I20260926 19:30:21.381150 15889 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:21.381369 15889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:21.387058 15889 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:30:21.390033 15904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:21.391486 15889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:21.391667 15889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
format_stamp: "Formatted at 2026-09-26 19:30:18 on dist-test-slave-9wb0"
server_key: "ffdbe1bd8642b157d5303a5c31844d14"
server_key_iv: "b57b906923b618cf0eddc70ba1702489"
server_key_version: "encryptionkey@0"
I20260926 19:30:21.392642 15889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:21.412446 15889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:21.416468 15889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:21.416756 15889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:21.417907 15889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:21.419211 15889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:21.419338 15889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:21.419479 15889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:21.419569 15889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:21.452628 15889 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:45811
I20260926 19:30:21.452684 16016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:45811 every 8 connection(s)
I20260926 19:30:21.454198 15889 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:21.464272 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 15889
Sep 26 19:30:21 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451021, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:21.475389 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.456802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:39747 (local address 127.1.39.62:44615)
0926 19:30:21.457003 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.457007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.458124 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 19:30:21.459949 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.459953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.460023 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.460130 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:21.461823 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.462516 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.464051 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.464525 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.464624 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.468460 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:21.468467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:21.468472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:21.468514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:21.472002 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.472816 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.472820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.472822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.472941 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.473468 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.473474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.473478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.473855 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:21.473975 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.474752 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.474983 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:30:21.476091 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.457242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:39747)
0926 19:30:21.457960 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.457996 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.459684 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.460362 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.460371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.460969 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:21.461667 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.461684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.462647 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.462650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.463910 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.463925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.464763 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.464769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.465016 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.465842 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:21.465890 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:21.468255 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:21.472150 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:21.472157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:21.472169 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:21.472694 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:21.473055 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:21.473060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:21.473064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:21.473339 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:21.473986 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:21.473997 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:21.474376 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.474597 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.474976 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":267,"threads_started":1}
I20260926 19:30:21.477711 16017 heartbeater.cc:347] Connected to a master server at 127.1.39.62:44615
I20260926 19:30:21.478047 16017 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:21.479027 16017 heartbeater.cc:511] Master 127.1.39.62:44615 requested a full tablet report, sending...
I20260926 19:30:21.480718 15569 ts_manager.cc:194] Registered new tserver with Master: d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
I20260926 19:30:21.481827 15569 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:39747
I20260926 19:30:21.482710 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.465409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39938 (local address 127.1.39.62:44615)
0926 19:30:21.466093 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.466099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.467457 (+ 1358us) server_negotiation.cc:407] Connection header received
0926 19:30:21.467640 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.467648 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.467733 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.467847 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:21.469730 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.470537 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.472495 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.473131 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.473251 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.476325 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:21.476333 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:21.476338 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:21.476376 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:21.479743 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.480584 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.480590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.480593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.480723 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.481111 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.481114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.481116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.481514 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:21.481747 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.482077 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.482293 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":412,"threads_started":1}
I20260926 19:30:21.485247 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:30:21.491827 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.486124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.1.39.62:44615)
0926 19:30:21.486288 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.486292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.486764 (+ 472us) server_negotiation.cc:407] Connection header received
0926 19:30:21.486921 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.486924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.487045 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.487159 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:30:21.487587 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.488263 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.489758 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.490208 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.490300 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.490737 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:30:21.491349 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:30:21.491461 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.491567 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.491639 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:30:21.492698 15569 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:30:21.500149 15569 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:30:21.526223 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.512293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42133 (local address 127.0.0.1:52442)
0926 19:30:21.514288 (+ 1995us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.514337 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.514541 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.516777 (+ 2236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.516788 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.516831 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:21.517384 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.517401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.519848 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.519856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.522963 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.522978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.525184 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.525192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.525462 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.525529 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.525638 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.525858 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1666,"thread_start_us":1508,"threads_started":1}
I20260926 19:30:21.527006 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.512458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.1.39.2:42133)
0926 19:30:21.516029 (+ 3571us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.516039 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.516073 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:30:21.516197 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.516206 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.516489 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.517674 (+ 1185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:21.517961 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.519700 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.523163 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.524699 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.524818 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.525023 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.526505 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.526694 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3382,"thread_start_us":3896,"threads_started":1}
I20260926 19:30:21.532125 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.512542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:45811 (local address 127.0.0.1:51172)
0926 19:30:21.515389 (+ 2847us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.515408 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.515524 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.524271 (+ 8747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.524276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.524304 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:21.524664 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.524677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.527093 (+ 2416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.527099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.529365 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.529381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.531293 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.531299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.531479 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.531517 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.531583 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.531725 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2725,"mutex_wait_us":40,"thread_start_us":2581,"threads_started":1}
I20260926 19:30:21.531793 15821 tablet_service.cc:1467] Processing CreateTablet for tablet f55e008f6ba44053a30127fac757b894 (DEFAULT_TABLE table=test-table [id=174b3bf496fb4a72b79526169a6aab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:21.533031 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.514698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34454)
0926 19:30:21.523880 (+ 9182us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.523900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.524013 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.524907 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.524912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.524936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:21.525398 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.525409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.527503 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.527510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.529902 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.529917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.532058 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.532064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.532350 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.532383 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.532464 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.532597 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9069,"mutex_wait_us":123,"thread_start_us":8158,"threads_started":1}
I20260926 19:30:21.533260 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.513367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.1.39.3:45811)
0926 19:30:21.523559 (+ 10192us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.523564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.523592 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:30:21.523715 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.523725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.524009 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.524850 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:21.525114 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.526921 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.529624 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.531063 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.532458 (+ 1395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.532701 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.532770 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.532933 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10045,"thread_start_us":9883,"threads_started":1}
I20260926 19:30:21.533725 15821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55e008f6ba44053a30127fac757b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:21.534029 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.515130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34454 (local address 127.1.39.1:35227)
0926 19:30:21.522763 (+ 7633us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.522776 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.524146 (+ 1370us) server_negotiation.cc:407] Connection header received
0926 19:30:21.524251 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.524257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.524467 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.524674 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:21.525620 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.527344 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.530095 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.531529 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.531654 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.531893 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.533258 (+ 1365us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.533531 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7403,"thread_start_us":7267,"threads_started":1}
I20260926 19:30:21.537075 15952 tablet_service.cc:1467] Processing CreateTablet for tablet f55e008f6ba44053a30127fac757b894 (DEFAULT_TABLE table=test-table [id=174b3bf496fb4a72b79526169a6aab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:21.537307 15689 tablet_service.cc:1467] Processing CreateTablet for tablet f55e008f6ba44053a30127fac757b894 (DEFAULT_TABLE table=test-table [id=174b3bf496fb4a72b79526169a6aab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:21.538522 15952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55e008f6ba44053a30127fac757b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:21.538527 15689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55e008f6ba44053a30127fac757b894. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:21.546603 16033 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Bootstrap starting.
I20260926 19:30:21.549832 16033 tablet_bootstrap.cc:654] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:21.551890 16033 log.cc:824] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:21.554433 16035 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Bootstrap starting.
I20260926 19:30:21.555073 16033 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: No bootstrap required, opened a new log
I20260926 19:30:21.555392 16033 ts_tablet_manager.cc:1397] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 19:30:21.556252 16036 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Bootstrap starting.
I20260926 19:30:21.558396 16035 tablet_bootstrap.cc:654] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:21.559068 16036 tablet_bootstrap.cc:654] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:21.559639 16035 log.cc:824] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:21.560135 16036 log.cc:824] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:21.563264 16035 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: No bootstrap required, opened a new log
I20260926 19:30:21.563380 15754 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:21.563560 16035 ts_tablet_manager.cc:1397] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:30:21.563901 16036 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: No bootstrap required, opened a new log
I20260926 19:30:21.564201 16036 ts_tablet_manager.cc:1397] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260926 19:30:21.566588 16033 raft_consensus.cc:348] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.567039 16033 raft_consensus.cc:374] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:21.567145 16033 raft_consensus.cc:729] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf8d703f4be4616ae26223644903e41, State: Initialized, Role: FOLLOWER
I20260926 19:30:21.567860 16033 consensus_queue.cc:260] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.574106 15886 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:21.573946 16036 raft_consensus.cc:348] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.574400 16036 raft_consensus.cc:374] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:21.574502 16036 raft_consensus.cc:729] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fd5e667ac445c0a771cabc369e0852, State: Initialized, Role: FOLLOWER
I20260926 19:30:21.575098 16036 consensus_queue.cc:260] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.575624 16035 raft_consensus.cc:348] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.576030 16035 raft_consensus.cc:374] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:21.576138 16035 raft_consensus.cc:729] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d100a3bced8e4fbc8b0aa0a151afe0c3, State: Initialized, Role: FOLLOWER
I20260926 19:30:21.576293 16033 ts_tablet_manager.cc:1428] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Time spent starting tablet: real 0.021s user 0.010s sys 0.011s
I20260926 19:30:21.576838 16035 consensus_queue.cc:260] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.577833 16036 ts_tablet_manager.cc:1428] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260926 19:30:21.578903 16017 heartbeater.cc:503] Master 127.1.39.62:44615 was elected leader, sending a full tablet report...
I20260926 19:30:21.580628 16035 ts_tablet_manager.cc:1428] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260926 19:30:21.660070 16040 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:21.660441 16040 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.662434 16040 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
I20260926 19:30:21.669976 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.662836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:54309 (local address 127.1.39.2:42133)
0926 19:30:21.663063 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.663071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.663809 (+ 738us) server_negotiation.cc:407] Connection header received
0926 19:30:21.664041 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.664048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.664146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.664238 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:21.665064 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.666232 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.668145 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.668877 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.669008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.669139 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.669572 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.669705 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:30:21.669976 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.662924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:44827 (local address 127.1.39.3:45811)
0926 19:30:21.663150 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.663158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.663987 (+ 829us) server_negotiation.cc:407] Connection header received
0926 19:30:21.664271 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.664276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.664352 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.664453 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:21.665123 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.666230 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.668107 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.668906 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.669058 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.669172 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.669577 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.669714 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:30:21.670276 16042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.662836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42133 (local address 127.1.39.1:54309)
0926 19:30:21.663696 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.663712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.663895 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.664310 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.664322 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.664350 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:21.664894 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.664908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.666368 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.666372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.667954 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.667965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.669128 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.669135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.669384 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.669441 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.669543 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.669684 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":98,"spinlock_wait_cycles":27904,"thread_start_us":485,"threads_started":1}
I20260926 19:30:21.670322 16043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.662946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:45811 (local address 127.1.39.1:44827)
0926 19:30:21.663880 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.663898 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:21.664097 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:21.664484 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:21.664492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:21.664528 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:21.664906 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.664915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.666428 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.666431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.667946 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:21.667958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.669167 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.669176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:21.669377 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.669442 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 19:30:21.669548 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 19:30:21.669686 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":54,"thread_start_us":549,"threads_started":1}
I20260926 19:30:21.671289 15841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55e008f6ba44053a30127fac757b894" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf8d703f4be4616ae26223644903e41" is_pre_election: true
I20260926 19:30:21.671343 15972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55e008f6ba44053a30127fac757b894" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" is_pre_election: true
I20260926 19:30:21.671788 15841 raft_consensus.cc:2454] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 0.
I20260926 19:30:21.671788 15972 raft_consensus.cc:2454] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 0.
I20260926 19:30:21.672465 15645 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852, abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3; no voters:
I20260926 19:30:21.672870 16040 raft_consensus.cc:2781] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:30:21.673012 16040 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:30:21.673146 16040 raft_consensus.cc:3037] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:21.675385 16040 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.676074 16040 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 election: Requested vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
I20260926 19:30:21.676481 15841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55e008f6ba44053a30127fac757b894" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abf8d703f4be4616ae26223644903e41"
I20260926 19:30:21.676481 15972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55e008f6ba44053a30127fac757b894" candidate_uuid: "60fd5e667ac445c0a771cabc369e0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
I20260926 19:30:21.676698 15972 raft_consensus.cc:3037] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:21.676698 15841 raft_consensus.cc:3037] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:21.679996 15972 raft_consensus.cc:2454] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 1.
I20260926 19:30:21.679996 15841 raft_consensus.cc:2454] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60fd5e667ac445c0a771cabc369e0852 in term 1.
I20260926 19:30:21.680405 15645 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852, abf8d703f4be4616ae26223644903e41; no voters:
I20260926 19:30:21.680699 16040 raft_consensus.cc:2781] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:21.681155 16040 raft_consensus.cc:686] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 LEADER]: Becoming Leader. State: Replica: 60fd5e667ac445c0a771cabc369e0852, State: Running, Role: LEADER
I20260926 19:30:21.681692 16040 consensus_queue.cc:237] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:21.686189 15569 catalog_manager.cc:5495] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "60fd5e667ac445c0a771cabc369e0852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } health_report { overall_health: HEALTHY } } }
W20260926 19:30:21.705768 16018 tablet.cc:2369] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:21.713514 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.708251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34466 (local address 127.1.39.1:35227)
0926 19:30:21.708452 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.708457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.708488 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:30:21.708666 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.708670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.708746 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.708860 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:30:21.709203 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.709993 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.711397 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.711795 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.711884 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.712350 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:30:21.712852 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:30:21.712965 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.713108 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.713201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:30:21.721577 15972 raft_consensus.cc:1261] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Refusing update from remote peer 60fd5e667ac445c0a771cabc369e0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:21.721575 15841 raft_consensus.cc:1261] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Refusing update from remote peer 60fd5e667ac445c0a771cabc369e0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:21.722682 16044 consensus_queue.cc:1035] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:21.723034 16040 consensus_queue.cc:1035] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:21.747740 15569 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:21.748034 15569 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:21.751281 15569 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Sending DeleteTablet for 3 replicas of tablet f55e008f6ba44053a30127fac757b894
I20260926 19:30:21.752501 15952 tablet_service.cc:1514] Processing DeleteTablet for tablet f55e008f6ba44053a30127fac757b894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:21 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:51172
I20260926 19:30:21.752475 15821 tablet_service.cc:1514] Processing DeleteTablet for tablet f55e008f6ba44053a30127fac757b894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:21 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:52442
I20260926 19:30:21.752758 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15606
I20260926 19:30:21.753947 15689 tablet_service.cc:1514] Processing DeleteTablet for tablet f55e008f6ba44053a30127fac757b894 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:21 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:34454
I20260926 19:30:21.776868 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.754192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.1.39.1:35227)
0926 19:30:21.754376 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.754381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.754398 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:21.754459 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.754474 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.754580 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.754713 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:21.757019 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.758525 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.761051 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.762006 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.762151 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.766635 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:21.766655 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:21.766681 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:21.766748 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:21.771742 (+ 4994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.772862 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.772876 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.772882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.773092 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.773641 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.773649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.773654 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.774209 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:21.774547 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.776021 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.776332 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:30:21.777818 15749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:21.778489 16056 tablet_replica.cc:330] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: stopping tablet replica
I20260926 19:30:21.779121 16056 raft_consensus.cc:2229] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:21.778512 16055 tablet_replica.cc:330] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: stopping tablet replica
I20260926 19:30:21.779881 16056 raft_consensus.cc:2258] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:21.780349 16055 raft_consensus.cc:2229] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:21.780918 16055 raft_consensus.cc:2258] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:21.782497 16056 ts_tablet_manager.cc:1905] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:21.784492 16055 ts_tablet_manager.cc:1905] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:21.787662 16056 ts_tablet_manager.cc:1918] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:21.787837 16056 log.cc:1196] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/f55e008f6ba44053a30127fac757b894
I20260926 19:30:21.788308 16056 ts_tablet_manager.cc:1939] T f55e008f6ba44053a30127fac757b894 P abf8d703f4be4616ae26223644903e41: Deleting consensus metadata
I20260926 19:30:21.790331 15556 catalog_manager.cc:4841] TS abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): tablet f55e008f6ba44053a30127fac757b894 (table test-table [id=174b3bf496fb4a72b79526169a6aab59]) successfully deleted
I20260926 19:30:21.791929 16055 ts_tablet_manager.cc:1918] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:21.792105 16055 log.cc:1196] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/f55e008f6ba44053a30127fac757b894
I20260926 19:30:21.792701 16055 ts_tablet_manager.cc:1939] T f55e008f6ba44053a30127fac757b894 P d100a3bced8e4fbc8b0aa0a151afe0c3: Deleting consensus metadata
I20260926 19:30:21.794770 15554 catalog_manager.cc:4841] TS d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): tablet f55e008f6ba44053a30127fac757b894 (table test-table [id=174b3bf496fb4a72b79526169a6aab59]) successfully deleted
I20260926 19:30:21.878228 16057 tablet_replica.cc:330] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: stopping tablet replica
I20260926 19:30:21.878722 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15606
I20260926 19:30:21.878822 16057 raft_consensus.cc:2229] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:30:21.898875 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15758
W20260926 19:30:21.898895 15556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.39.1:35227 (error 108)
I20260926 19:30:21.900775 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.900249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34482)
0926 19:30:21.900446 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.900566 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 19:30:21.901015 15556 catalog_manager.cc:4584] TS 60fd5e667ac445c0a771cabc369e0852 (127.1.39.1:35227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f55e008f6ba44053a30127fac757b894: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
I20260926 19:30:21.912881 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.899786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.1.39.2:42133)
0926 19:30:21.899974 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:21.899980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:21.900023 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:30:21.900237 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:21.900242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:21.900340 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:21.900448 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:21.902151 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.902921 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.904180 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:21.904582 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:21.904678 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:21.907457 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:21.907470 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:21.907486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:21.907523 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:21.910406 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.911105 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.911112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.911114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.911232 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:21.911535 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:21.911538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:21.911540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:21.911895 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:21.912116 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:21.912352 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 19:30:21.912508 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:30:21.913637 15881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:21.954677 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.954174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34484)
0926 19:30:21.954390 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.954512 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 19:30:21.955698 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:21.955354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34494)
0926 19:30:21.955517 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:21.955583 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 19:30:22.005369 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15758
I20260926 19:30:22.016453 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:22.015912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34496)
0926 19:30:22.016150 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:22.016258 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 19:30:22.017732 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:22.017219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34506)
0926 19:30:22.017445 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:22.017552 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:30:22.024772 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15889
I20260926 19:30:22.038439 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:22.025764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.1.39.3:45811)
0926 19:30:22.025974 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:22.025978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:22.025993 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:30:22.026045 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:22.026048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:22.026112 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:22.026222 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:22.027629 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:22.028346 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:22.029700 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:22.030092 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:22.030205 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:22.032629 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:22.032643 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:22.032660 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:22.032697 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:22.035360 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:22.036052 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:22.036066 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:22.036069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:22.036211 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:22.036581 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:22.036599 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:22.036602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:22.036942 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:22.037236 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:22.037855 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 19:30:22.038088 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:30:22.039045 16012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:22.099407 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:22.098810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34516)
0926 19:30:22.099066 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:22.099186 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 19:30:22.100656 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:22.100174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:35227 (local address 127.0.0.1:34522)
0926 19:30:22.100392 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:22.100493 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:35227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 19:30:22.130787 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15889
I20260926 19:30:22.148262 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 15536
I20260926 19:30:22.149569 15599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:22.265437 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 15536
I20260926 19:30:22.289130 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--webserver_interface=127.1.39.62
--webserver_port=42871
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:44615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:22.627338 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:22.627722 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:22.627875 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:22.635608 16063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:22.635767 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:22.635880 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:22.635980 16063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:22.636070 16063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:22.646374 16063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:44615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:44615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=42871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:22.647817 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:22.649135 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:22.661561 16068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:22.662708 16063 server_base.cc:1034] running on GCE node
W20260926 19:30:22.665510 16071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:22.665531 16069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:22.666553 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:22.668103 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:22.669382 16063 hybrid_clock.cc:648] HybridClock initialized: now 1790451022669336 us; error 38 us; skew 500 ppm
I20260926 19:30:22.669742 16063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:22 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451022, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:22.672915 16063 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:22.674479 16063 webserver.cc:460] Webserver started at http://127.1.39.62:42871/ using document root <none> and password file <none>
I20260926 19:30:22.675144 16063 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:22.675246 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:22.680965 16063 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 19:30:22.684042 16078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:22.685542 16063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 19:30:22.685698 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cc2eca70614c4b4084298052bedfae36"
format_stamp: "Formatted at 2026-09-26 19:30:16 on dist-test-slave-9wb0"
server_key: "24c69d2997f387d99b14e2b71407447c"
server_key_iv: "0f3ed9639836792edaefa72d0c2c065d"
server_key_version: "encryptionkey@0"
I20260926 19:30:22.686656 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:22.701036 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:22.705273 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:22.705598 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:22.724833 16063 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:44615
I20260926 19:30:22.724881 16129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:44615 every 8 connection(s)
I20260926 19:30:22.726378 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:22.733240 16130 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:30:22.733745 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16063
I20260926 19:30:22.734565 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:35227
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=42703
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:22.736781 16130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap starting.
I20260926 19:30:22.764616 16130 log.cc:824] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:22.785079 16130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:30:22.785842 16130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36: Bootstrap complete.
I20260926 19:30:22.799244 16130 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:22.800196 16130 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Initialized, Role: FOLLOWER
I20260926 19:30:22.800952 16130 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:22.801195 16130 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:22.801342 16130 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:22.801534 16130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:30:22.805444 16130 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:22.805961 16130 leader_election.cc:302] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc2eca70614c4b4084298052bedfae36; no voters:
I20260926 19:30:22.806586 16130 leader_election.cc:288] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 19:30:22.806792 16134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 19:30:22.807847 16134 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [term 3 LEADER]: Becoming Leader. State: Replica: cc2eca70614c4b4084298052bedfae36, State: Running, Role: LEADER
I20260926 19:30:22.808393 16134 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } }
I20260926 19:30:22.808794 16130 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:22.812242 16136 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc2eca70614c4b4084298052bedfae36. Latest consensus state: current_term: 3 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:22.812742 16136 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:22.813498 16135 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cc2eca70614c4b4084298052bedfae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc2eca70614c4b4084298052bedfae36" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 44615 } } }
I20260926 19:30:22.813823 16135 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc2eca70614c4b4084298052bedfae36 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:22.822542 16140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:22.834468 16140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:22.837070 16140 catalog_manager.cc:1259] Loaded cluster ID: 21ffe4829dc349dab09dd414decd12bb
I20260926 19:30:22.837319 16140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 19:30:22.841572 16140 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe672a989ad google::LogMessage::Fail() at ??:0
@ 0x7fe672a9b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe672a984de google::LogMessage::Flush() at ??:0
@ 0x7fe672a99a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6884cb512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe6884c8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe688550610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe688550439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe6756e4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe6756e791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe6756e7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe6756c2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe67899c6db start_thread at ??:0
@ 0x7fe67193261f clone at ??:0
W20260926 19:30:23.121207 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:23.121618 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:23.121760 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:23.129055 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:23.129168 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:23.129333 16132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:23.139061 16132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=42703
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:23.140297 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:23.141604 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:23.154662 16172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:23.154765 16174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:23.154891 16171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:23.155499 16132 server_base.cc:1034] running on GCE node
I20260926 19:30:23.156078 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:23.156998 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:23.158219 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790451023158194 us; error 34 us; skew 500 ppm
I20260926 19:30:23.158527 16132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:23 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:23.161636 16132 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:23.162976 16132 webserver.cc:460] Webserver started at http://127.1.39.1:42703/ using document root <none> and password file <none>
I20260926 19:30:23.163630 16132 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:23.163729 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:23.168795 16132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:30:23.171552 16181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:23.172866 16132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:30:23.173033 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "60fd5e667ac445c0a771cabc369e0852"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "bcdcf94d05758d9a07ec135a971bc258"
server_key_iv: "a1bbc85ef97fd6b9df2e44f958095d5f"
server_key_version: "encryptionkey@0"
I20260926 19:30:23.174054 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:23.193173 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:23.197289 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:23.197597 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:23.198611 16132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:23.200342 16188 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 19:30:23.205936 16132 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 19:30:23.206048 16132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 19:30:23.206198 16132 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 19:30:23.209427 16132 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 19:30:23.209503 16132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:23.209848 16188 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Bootstrap starting.
I20260926 19:30:23.236982 16188 log.cc:824] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:23.249186 16188 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:30:23.249902 16188 tablet_bootstrap.cc:492] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Bootstrap complete.
I20260926 19:30:23.251724 16188 ts_tablet_manager.cc:1397] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Time spent bootstrapping tablet: real 0.042s user 0.035s sys 0.004s
I20260926 19:30:23.254352 16132 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:35227
I20260926 19:30:23.254417 16295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:35227 every 8 connection(s)
I20260926 19:30:23.255635 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:23.260833 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.258497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:49267)
0926 19:30:23.259430 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.259655 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":372,"threads_started":1}
I20260926 19:30:23.260185 16188 raft_consensus.cc:348] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:23.261498 16188 raft_consensus.cc:729] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fd5e667ac445c0a771cabc369e0852, State: Initialized, Role: FOLLOWER
W20260926 19:30:23.261547 16296 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:44615 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:44615: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
I20260926 19:30:23.262141 16188 consensus_queue.cc:260] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:23.262524 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.262150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:44207)
0926 19:30:23.262295 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.262361 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:30:23.263399 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.263048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:56163)
0926 19:30:23.263187 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.263274 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:30:23.263649 16296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:30:23.263775 16188 ts_tablet_manager.cc:1428] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 19:30:23.264315 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.264014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:55701)
0926 19:30:23.264152 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.264221 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:30:23.264364 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16132
I20260926 19:30:23.265018 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:42133
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=40551
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:23.265134 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.264832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:33113)
0926 19:30:23.264962 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.265030 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:30:23.598788 16300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:23.599129 16300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:23.599301 16300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:23.606542 16300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:23.606712 16300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:23.606894 16300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:23.616859 16300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:42133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=40551
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:23.618230 16300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:23.619529 16300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:23.632418 16305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:23.632500 16308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:23.632462 16306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:23.633641 16300 server_base.cc:1034] running on GCE node
I20260926 19:30:23.634364 16300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:23.635344 16300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:23.636574 16300 hybrid_clock.cc:648] HybridClock initialized: now 1790451023636523 us; error 47 us; skew 500 ppm
I20260926 19:30:23.636893 16300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:23 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:23.639945 16300 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:23.641458 16300 webserver.cc:460] Webserver started at http://127.1.39.2:40551/ using document root <none> and password file <none>
I20260926 19:30:23.642179 16300 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:23.642335 16300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:23.647475 16300 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 19:30:23.650204 16315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:23.651523 16300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:23.651710 16300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "abf8d703f4be4616ae26223644903e41"
format_stamp: "Formatted at 2026-09-26 19:30:17 on dist-test-slave-9wb0"
server_key: "5be17414411667a59e9e8447318418d8"
server_key_iv: "b9d3f37a8b8ec2dffc0b38d938119514"
server_key_version: "encryptionkey@0"
I20260926 19:30:23.652733 16300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:23.674388 16300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:23.678534 16300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:23.678822 16300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:23.679821 16300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:23.681124 16300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:23.681252 16300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:23.681409 16300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:23.681500 16300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:23.715709 16300 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:42133
I20260926 19:30:23.715801 16427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:42133 every 8 connection(s)
I20260926 19:30:23.717206 16300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:23.719743 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16300
I20260926 19:30:23.720489 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:45811
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:44615
--webserver_port=33691
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:30:23.722397 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.720492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:48651)
0926 19:30:23.721272 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.721547 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":309,"threads_started":1}
W20260926 19:30:23.723047 16428 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:44615 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:44615: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
I20260926 19:30:23.723948 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.723612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:41661)
0926 19:30:23.723772 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.723849 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 19:30:23.724762 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:23.724448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:60915)
0926 19:30:23.724584 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:23.724665 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 19:30:23.725028 16428 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 19:30:24.054632 16431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:24.054917 16431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:24.055011 16431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:24.062490 16431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:24.062613 16431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:24.062727 16431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:24.072469 16431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:45811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=33691
--tserver_master_addrs=127.1.39.62:44615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:24.073841 16431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:24.075094 16431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:24.088615 16436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:24.088656 16437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:24.088709 16439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:24.089524 16431 server_base.cc:1034] running on GCE node
I20260926 19:30:24.090159 16431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:24.091208 16431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:24.092439 16431 hybrid_clock.cc:648] HybridClock initialized: now 1790451024092391 us; error 65 us; skew 500 ppm
I20260926 19:30:24.092761 16431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:24 dist-test-slave-9wb0 krb5kdc[14985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451024, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:24.096091 16431 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:24.097591 16431 webserver.cc:460] Webserver started at http://127.1.39.3:33691/ using document root <none> and password file <none>
I20260926 19:30:24.098279 16431 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:24.098423 16431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:24.103593 16431 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 19:30:24.106287 16446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:24.107491 16431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 19:30:24.107673 16431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3"
format_stamp: "Formatted at 2026-09-26 19:30:18 on dist-test-slave-9wb0"
server_key: "ffdbe1bd8642b157d5303a5c31844d14"
server_key_iv: "b57b906923b618cf0eddc70ba1702489"
server_key_version: "encryptionkey@0"
I20260926 19:30:24.108649 16431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:24.121711 16431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:24.125980 16431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:24.126271 16431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:24.127260 16431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:24.128582 16431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:24.128708 16431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:24.128844 16431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:24.128931 16431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:24.163372 16431 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:45811
I20260926 19:30:24.163460 16558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:45811 every 8 connection(s)
I20260926 19:30:24.164806 16431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:24.169723 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.167834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:51493)
0926 19:30:24.168576 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.168771 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
W20260926 19:30:24.170532 16559 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:44615 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:44615: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
I20260926 19:30:24.171561 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.171182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:49223)
0926 19:30:24.171365 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.171442 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 19:30:24.172561 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.172214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:40383)
0926 19:30:24.172374 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.172454 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 19:30:24.172899 16559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:30:24.175278 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16431
I20260926 19:30:24.266816 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.265997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:33681)
0926 19:30:24.266501 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.266621 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":359,"threads_started":1}
I20260926 19:30:24.605393 16564 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:24.605726 16564 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:24.607312 16564 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
Sep 26 19:30:24 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
Sep 26 19:30:24 dist-test-slave-9wb0 krb5kdc[14985](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451023, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:30:24.632933 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.608026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:45215 (local address 127.1.39.3:45811)
0926 19:30:24.610145 (+ 2119us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:24.610155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:24.610200 (+ 45us) server_negotiation.cc:407] Connection header received
0926 19:30:24.611882 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:24.611890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:24.612281 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:24.612613 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:24.615129 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.617016 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.619750 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.620759 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.620950 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:24.625906 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:24.625914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:24.625930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:24.625967 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:24.629051 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630018 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:24.630025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:24.630030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:24.630202 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630667 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:24.630676 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:24.630678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:24.631175 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:24.631374 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:24.632126 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 19:30:24.632488 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1963,"thread_start_us":1868,"threads_started":1}
I20260926 19:30:24.632952 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.607723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:45811 (local address 127.1.39.1:45215)
0926 19:30:24.609538 (+ 1815us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.609566 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:24.609720 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:24.612748 (+ 3028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:24.612758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:24.613830 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:24.614901 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:24.614930 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.617249 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.617253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:24.619589 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:24.619606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.621006 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.621018 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:24.621408 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:24.622501 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:24.622546 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:24.625582 (+ 3036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:24.629217 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:24.629226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:24.629243 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:24.629871 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:24.630319 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:24.630328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:24.630542 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:24.631314 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:24.631322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:24.631651 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 19:30:24.631920 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 19:30:24.632497 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1655,"mutex_wait_us":463,"spinlock_wait_cycles":56320,"thread_start_us":1433,"threads_started":1}
I20260926 19:30:24.632947 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.607922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:40127 (local address 127.1.39.2:42133)
0926 19:30:24.609932 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:24.609948 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:24.609996 (+ 48us) server_negotiation.cc:407] Connection header received
0926 19:30:24.611667 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:24.611676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:24.612265 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:24.612613 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:24.615129 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.617016 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.619750 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.620759 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.620950 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:24.626119 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:24.626126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:24.626138 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:24.626175 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:24.629162 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630019 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:24.630026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:24.630031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:24.630202 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630692 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:24.630697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:24.630701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:24.631180 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:24.631376 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:24.632159 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 19:30:24.632523 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1793,"thread_start_us":1682,"threads_started":1}
I20260926 19:30:24.633390 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.607661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:42133 (local address 127.1.39.1:40127)
0926 19:30:24.607983 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.608026 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:24.608673 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:24.612747 (+ 4074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:24.612757 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:24.613830 (+ 1073us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:24.614901 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:24.614930 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.617245 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.617251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:24.619589 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:24.619609 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:24.621027 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:24.621033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:24.621403 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:24.622531 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:24.622554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:24.625849 (+ 3295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:24.629332 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:24.629342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:24.629346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:24.629870 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:24.630319 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:24.630325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:24.630328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:24.630554 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:24.631314 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:24.631323 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:24.631652 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 19:30:24.631922 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 19:30:24.632661 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":620,"spinlock_wait_cycles":36352}
W20260926 19:30:24.637043 16183 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
W20260926 19:30:24.637370 16185 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
I20260926 19:30:24.637562 16185 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852; no voters: abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3
I20260926 19:30:24.638008 16564 raft_consensus.cc:2726] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:30:24.726444 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:24.725613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:38623)
0926 19:30:24.726068 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:24.726189 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":214,"threads_started":1}
I20260926 19:30:25.174268 16569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:25.173468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:54927)
0926 19:30:25.173925 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:25.174059 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":211,"threads_started":1}
I20260926 19:30:25.268474 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:25.267797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:36395)
0926 19:30:25.268240 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:25.268318 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":255,"threads_started":1}
I20260926 19:30:25.727964 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:25.727288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:51481)
0926 19:30:25.727730 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:25.727810 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":217,"threads_started":1}
I20260926 19:30:26.175913 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:26.175177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:52579)
0926 19:30:26.175648 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:26.175737 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260926 19:30:26.267333 16573 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:26.267542 16573 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:26.268234 16573 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
W20260926 19:30:26.269351 16185 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
W20260926 19:30:26.269538 16183 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
I20260926 19:30:26.269742 16183 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852; no voters: abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3
I20260926 19:30:26.270056 16573 raft_consensus.cc:2726] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:30:26.270438 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:26.269409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:47815)
0926 19:30:26.270118 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:26.270253 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":324,"threads_started":1}
I20260926 19:30:26.729725 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:26.728902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:37229)
0926 19:30:26.729403 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:26.729488 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260926 19:30:27.177964 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:27.176799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:46919)
0926 19:30:27.177493 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:27.177786 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260926 19:30:27.272068 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:27.271342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:42779)
0926 19:30:27.271814 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:27.271905 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":265,"threads_started":1}
I20260926 19:30:27.731418 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:27.730693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:44551)
0926 19:30:27.731145 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:27.731234 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":237,"threads_started":1}
I20260926 19:30:27.885217 16579 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:27.885542 16579 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:27.886238 16579 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
W20260926 19:30:27.887305 16185 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
W20260926 19:30:27.887529 16183 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
I20260926 19:30:27.887705 16183 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852; no voters: abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3
I20260926 19:30:27.888010 16579 raft_consensus.cc:2726] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:30:28.179652 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:28.178933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:49509)
0926 19:30:28.179370 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:28.179450 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":207,"threads_started":1}
I20260926 19:30:28.274199 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:28.273339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:38967)
0926 19:30:28.273867 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:28.273977 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":232,"threads_started":1}
I20260926 19:30:28.733227 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:28.732489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:47687)
0926 19:30:28.732964 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:28.733060 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":271,"threads_started":1}
I20260926 19:30:29.181488 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:29.180583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:38821)
0926 19:30:29.181154 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:29.181249 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":320,"threads_started":1}
I20260926 19:30:29.275818 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:29.275087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:35353)
0926 19:30:29.275566 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:29.275655 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260926 19:30:29.734948 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:29.734200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:56313)
0926 19:30:29.734680 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:29.734767 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260926 19:30:30.183111 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:30.182368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:37249)
0926 19:30:30.182851 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:30.182929 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260926 19:30:30.277740 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:30.276806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:38071)
0926 19:30:30.277278 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:30.277459 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
I20260926 19:30:30.736937 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:30.736008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:53281)
0926 19:30:30.736654 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:30.736740 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":294,"threads_started":1}
I20260926 19:30:31.185017 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:31.184076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:56223)
0926 19:30:31.184628 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:31.184733 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":250,"threads_started":1}
I20260926 19:30:31.279331 16590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:31.278664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:54551)
0926 19:30:31.279092 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:31.279177 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":230,"threads_started":1}
I20260926 19:30:31.738781 16591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:31.738018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:57577)
0926 19:30:31.738521 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:31.738603 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":288,"threads_started":1}
I20260926 19:30:32.163761 16592 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:32.164002 16592 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:32.164717 16592 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
W20260926 19:30:32.165699 16185 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
W20260926 19:30:32.165882 16183 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
I20260926 19:30:32.166045 16183 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852; no voters: abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3
I20260926 19:30:32.166308 16592 raft_consensus.cc:2726] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:30:32.186719 16593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:32.185999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:51935)
0926 19:30:32.186443 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:32.186519 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
I20260926 19:30:32.281076 16594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:32.280272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:57463)
0926 19:30:32.280779 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:32.280869 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":252,"threads_started":1}
I20260926 19:30:32.740629 16595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:32.739743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:55297)
0926 19:30:32.740304 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:32.740391 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
I20260926 19:30:33.188527 16596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:33.187803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:47161)
0926 19:30:33.188263 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:33.188343 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":221,"threads_started":1}
I20260926 19:30:33.282836 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:33.282077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:38433)
0926 19:30:33.282579 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:33.282663 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":249,"threads_started":1}
I20260926 19:30:33.742264 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:33.741553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:43131)
0926 19:30:33.741998 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:33.742079 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":243,"threads_started":1}
I20260926 19:30:34.190270 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:34.189522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:41197)
0926 19:30:34.189979 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:34.190059 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":247,"threads_started":1}
I20260926 19:30:34.284401 16600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:34.283673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:34813)
0926 19:30:34.284125 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:34.284206 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260926 19:30:34.744007 16601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:34.743217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:51739)
0926 19:30:34.743737 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:34.743826 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":301,"threads_started":1}
I20260926 19:30:35.192057 16602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:35.191267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:50217)
0926 19:30:35.191804 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:35.191896 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":263,"threads_started":1}
I20260926 19:30:35.286149 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:35.285413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:60933)
0926 19:30:35.285869 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:35.285954 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":238,"threads_started":1}
I20260926 19:30:35.745816 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:35.745017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:33393)
0926 19:30:35.745561 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:35.745645 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":327,"threads_started":1}
I20260926 19:30:36.193854 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:36.193009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:58109)
0926 19:30:36.193576 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:36.193661 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":316,"threads_started":1}
I20260926 19:30:36.287717 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:36.286985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:50123)
0926 19:30:36.287461 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:36.287541 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":256,"threads_started":1}
I20260926 19:30:36.747726 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:36.746766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:40045)
0926 19:30:36.747358 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:36.747491 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":267,"threads_started":1}
I20260926 19:30:37.195537 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:37.194845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:42787)
0926 19:30:37.195293 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:37.195373 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260926 19:30:37.289466 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:37.288591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:55545)
0926 19:30:37.289051 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:37.289138 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260926 19:30:37.749413 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:37.748618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:51605)
0926 19:30:37.749076 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:37.749154 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":231,"threads_started":1}
I20260926 19:30:37.833932 16611 raft_consensus.cc:482] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:37.834219 16611 raft_consensus.cc:504] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abf8d703f4be4616ae26223644903e41" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 42133 } } peers { permanent_uuid: "d100a3bced8e4fbc8b0aa0a151afe0c3" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 45811 } } peers { permanent_uuid: "60fd5e667ac445c0a771cabc369e0852" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 35227 } }
I20260926 19:30:37.834985 16611 leader_election.cc:288] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133), d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811)
W20260926 19:30:37.835937 16183 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d100a3bced8e4fbc8b0aa0a151afe0c3 (127.1.39.3:45811): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
W20260926 19:30:37.836097 16185 leader_election.cc:341] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer abf8d703f4be4616ae26223644903e41 (127.1.39.2:42133): Not found: Tablet not found: f55e008f6ba44053a30127fac757b894
I20260926 19:30:37.836215 16185 leader_election.cc:302] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60fd5e667ac445c0a771cabc369e0852; no voters: abf8d703f4be4616ae26223644903e41, d100a3bced8e4fbc8b0aa0a151afe0c3
I20260926 19:30:37.836443 16611 raft_consensus.cc:2726] T f55e008f6ba44053a30127fac757b894 P 60fd5e667ac445c0a771cabc369e0852 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:30:38.197474 16612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:38.196495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:54483)
0926 19:30:38.197031 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:38.197136 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":262,"threads_started":1}
I20260926 19:30:38.291667 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:38.290709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.1:49801)
0926 19:30:38.291327 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:38.291426 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":303,"threads_started":1}
I20260926 19:30:38.751557 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:38.750586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.2:51957)
0926 19:30:38.751238 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:38.751357 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":397,"threads_started":1}
I20260926 19:30:39.179783 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16132
I20260926 19:30:39.195772 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:39.180806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.1.39.1:35227)
0926 19:30:39.181436 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:39.181447 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:39.181477 (+ 30us) server_negotiation.cc:407] Connection header received
0926 19:30:39.181559 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:39.181567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:39.181831 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:39.182038 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:39.184209 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.185190 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.186561 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.187099 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.187210 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:39.189853 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:39.189864 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:39.189878 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:39.189912 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:39.192818 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.193581 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.193588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.193591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.193745 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.194194 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.194198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.194201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.194661 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:39.194920 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:39.195264 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 19:30:39.195490 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":261,"threads_started":1}
I20260926 19:30:39.197062 16291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:39.199591 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:39.198720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:44615 (local address 127.1.39.3:59089)
0926 19:30:39.199247 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:39.199357 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":263,"threads_started":1}
I20260926 19:30:39.286360 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16132
I20260926 19:30:39.305603 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16300
I20260926 19:30:39.319696 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:39.306432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.1.39.2:42133)
0926 19:30:39.306917 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:39.306921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:39.306938 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:39.306995 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:39.306999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:39.307073 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:39.307165 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:39.308434 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.309239 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.310573 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.311031 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.311138 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:39.313804 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:39.313810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:39.313813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:39.313848 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:39.316662 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.317438 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.317447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.317452 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.317619 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.318084 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.318090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.318094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.318480 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:39.318758 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:39.319077 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:30:39.319317 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":275,"threads_started":1}
I20260926 19:30:39.320405 16423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:39.405237 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16300
I20260926 19:30:39.423895 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16431
I20260926 19:30:39.437933 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:39.424737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.1.39.3:45811)
0926 19:30:39.425375 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:39.425380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:39.425397 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:39.425450 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:39.425453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:39.425521 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:39.425652 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:39.427041 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.427854 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.429238 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:39.429734 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:39.429872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:39.432453 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:39.432458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:39.432462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:39.432496 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:39.435105 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.435786 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.435794 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.435802 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.435937 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:39.436328 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:39.436333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:39.436337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:39.436613 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:39.436903 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:39.437228 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 19:30:39.437504 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":299,"threads_started":1}
I20260926 19:30:39.438686 16554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:39.523083 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16431
2026-09-26T19:30:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24483 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:30:39 dist-test-slave-9wb0 krb5kdc[16624](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:30:39 dist-test-slave-9wb0 krb5kdc[16624](info): set up 2 sockets
Sep 26 19:30:39 dist-test-slave-9wb0 krb5kdc[16624](info): commencing operation
krb5kdc: starting...
W20260926 19:30:40.665938 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:30:40 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:30:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:30:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:30:40.925221 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:41.271516 16637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:41.271906 16637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:41.272092 16637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:41.280146 16637 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:30:41.280289 16637 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:30:41.280385 16637 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:30:41.280488 16637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:30:41.280593 16637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:41.280695 16637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:41.280768 16637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:30:41.280846 16637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:30:41.291440 16637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:41.292841 16637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:41.294298 16637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:41.305859 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:41.305944 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:41.306034 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:41.307407 16637 server_base.cc:1034] running on GCE node
I20260926 19:30:41.308245 16637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:41.309846 16637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:41.311101 16637 hybrid_clock.cc:648] HybridClock initialized: now 1790451041311032 us; error 34 us; skew 500 ppm
I20260926 19:30:41.311497 16637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:41 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:41.314926 16637 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:30:41.316504 16637 webserver.cc:460] Webserver started at http://127.1.39.62:33857/ using document root <none> and password file <none>
I20260926 19:30:41.317375 16637 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:41.317524 16637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:41.317883 16637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:41.321170 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "a0f54fb07dcf400b8066ec39f99fa531"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0c28d91eff7117bcdb0035e3e8aa1364"
server_key_iv: "c9428010b230fc597f7effaa18706e0e"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.322125 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "a0f54fb07dcf400b8066ec39f99fa531"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0c28d91eff7117bcdb0035e3e8aa1364"
server_key_iv: "c9428010b230fc597f7effaa18706e0e"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.327931 16637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:30:41.331647 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:41.333029 16637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:30:41.333242 16637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "a0f54fb07dcf400b8066ec39f99fa531"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0c28d91eff7117bcdb0035e3e8aa1364"
server_key_iv: "c9428010b230fc597f7effaa18706e0e"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.333585 16637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:41.350250 16637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:41.354776 16637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:41.355086 16637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:41.375461 16637 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42859
I20260926 19:30:41.375486 16703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42859 every 8 connection(s)
I20260926 19:30:41.376940 16637 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:30:41.380669 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16637
I20260926 19:30:41.380712 16704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:41.380908 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:30:41.381692 1180 external_mini_cluster.cc:1428] Setting key 2602f334d55b3d96f12a1fc9c280394e
I20260926 19:30:41.394814 16704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap starting.
Sep 26 19:30:41 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:41.398723 16704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:41.399967 16704 log.cc:824] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:41.403241 16704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: No bootstrap required, opened a new log
I20260926 19:30:41.407612 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:41.383743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.1.39.62:42859)
0926 19:30:41.384592 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:41.384604 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:41.384647 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:30:41.386844 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:41.386867 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:41.387312 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:41.387672 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:41.389724 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:41.391160 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:41.393135 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:41.393896 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:41.394047 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:41.398779 (+ 4732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:41.398787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:41.398801 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:41.398865 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:41.402744 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:41.403694 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:41.403703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:41.403714 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:41.403908 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:41.404390 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:41.404398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:41.404402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:41.405049 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:41.405480 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:41.406016 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 19:30:41.406480 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":414,"threads_started":1}
I20260926 19:30:41.411809 16704 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:30:41.412168 16704 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:41.412278 16704 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Initialized, Role: FOLLOWER
I20260926 19:30:41.412869 16704 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:30:41.413120 16704 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:30:41.413266 16704 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:30:41.413534 16704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:41.415767 16704 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:30:41.416324 16704 leader_election.cc:302] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0f54fb07dcf400b8066ec39f99fa531; no voters:
I20260926 19:30:41.417022 16704 leader_election.cc:288] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:30:41.417141 16709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:41.418038 16709 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 LEADER]: Becoming Leader. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Running, Role: LEADER
I20260926 19:30:41.418594 16709 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:30:41.419312 16704 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:30:41.421216 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:30:41.421386 16711 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f54fb07dcf400b8066ec39f99fa531. Latest consensus state: current_term: 1 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:30:41.421643 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:41.421646 16711 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:30:41.422592 16716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:30:41.428274 16716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:30:41.437486 16716 catalog_manager.cc:1347] Generated new cluster ID: 3baba09097db4cf591317ac2ce6ebca1
I20260926 19:30:41.437665 16716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:30:41.449115 16716 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:30:41.451012 16716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:30:41.470454 16716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Generated new TSK 0
I20260926 19:30:41.471602 16716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:30:41.541420 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:41.871693 16730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:41.871964 16730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:41.872057 16730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:41.879197 16730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:41.879302 16730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:41.879415 16730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:30:41.888906 16730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:41.890230 16730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:41.891462 16730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:41.905085 16735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:41.905007 16736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:41.905059 16738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:41.906605 16730 server_base.cc:1034] running on GCE node
I20260926 19:30:41.907299 16730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:41.908242 16730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:41.909461 16730 hybrid_clock.cc:648] HybridClock initialized: now 1790451041909396 us; error 56 us; skew 500 ppm
I20260926 19:30:41.909818 16730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:41 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:41.913102 16730 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:30:41.914662 16730 webserver.cc:460] Webserver started at http://127.1.39.1:43967/ using document root <none> and password file <none>
I20260926 19:30:41.915362 16730 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:41.915469 16730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:41.915797 16730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:41.918824 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "5707388c209843ff9c56f27c103c75b0"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0af587ffad56a458139bd8b0abb79418"
server_key_iv: "d65699405e4383ddcbaaf6329bd32dff"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.919665 16730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "5707388c209843ff9c56f27c103c75b0"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0af587ffad56a458139bd8b0abb79418"
server_key_iv: "d65699405e4383ddcbaaf6329bd32dff"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.925354 16730 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:30:41.928893 16745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:41.930454 16730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:30:41.930629 16730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "5707388c209843ff9c56f27c103c75b0"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0af587ffad56a458139bd8b0abb79418"
server_key_iv: "d65699405e4383ddcbaaf6329bd32dff"
server_key_version: "encryptionkey@0"
I20260926 19:30:41.930831 16730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:41.952598 16730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:41.957242 16730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:41.957613 16730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:41.958626 16730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:41.959962 16730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:41.960041 16730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:41.960165 16730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:41.960242 16730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:41.993717 16730 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:46457
I20260926 19:30:41.993804 16857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:46457 every 8 connection(s)
I20260926 19:30:41.994997 16730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:30:41.996297 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16730
I20260926 19:30:41.996512 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:30:41.997378 1180 external_mini_cluster.cc:1428] Setting key 20dfadd5877c8e7239b1f29a819dbe32
Sep 26 19:30:42 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451041, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:42.020870 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:41.998479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:56445 (local address 127.1.39.62:42859)
0926 19:30:41.999005 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:41.999012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:41.999795 (+ 783us) server_negotiation.cc:407] Connection header received
0926 19:30:42.002461 (+ 2666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:42.002465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:42.002527 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:42.002613 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:42.004762 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.006202 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.008746 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.009509 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.009666 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:42.013888 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:42.013897 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:42.013905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:42.013952 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:42.017519 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:42.018228 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:42.018236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:42.018242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:42.018393 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:42.018801 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:42.018809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:42.018815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:42.019206 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:42.019351 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:42.020078 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 19:30:42.020345 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":262,"threads_started":1}
I20260926 19:30:42.021736 16861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:41.998527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:56445)
0926 19:30:41.999394 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:41.999432 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:42.002169 (+ 2737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:42.002878 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:42.002892 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:42.003688 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:42.004489 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:42.004517 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.006387 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.006396 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:42.008561 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:42.008579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.009739 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.009745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:42.010077 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:42.010863 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:42.010912 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:42.013651 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:42.017653 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:42.017660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:42.017672 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:42.018093 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:42.018508 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:42.018511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:42.018513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:42.018683 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:42.019355 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:42.019364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:42.019685 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 19:30:42.019915 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 19:30:42.020406 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":327,"threads_started":1}
I20260926 19:30:42.023353 16858 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:30:42.023694 16858 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:42.024461 16858 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
I20260926 19:30:42.026608 16669 ts_manager.cc:194] Registered new tserver with Master: 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457)
I20260926 19:30:42.028061 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:56445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:30:42.060329 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:42.394465 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:42.394820 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:42.394954 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:42.403834 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:42.403944 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:42.404057 16864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:30:42.413822 16864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:42.415277 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:42.416546 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:42.429363 16869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:42.429426 16870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:42.429567 16872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:42.430137 16864 server_base.cc:1034] running on GCE node
I20260926 19:30:42.430732 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:42.431666 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:42.432901 16864 hybrid_clock.cc:648] HybridClock initialized: now 1790451042432861 us; error 52 us; skew 500 ppm
I20260926 19:30:42.433213 16864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:42 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:42.436692 16864 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:30:42.438324 16864 webserver.cc:460] Webserver started at http://127.1.39.2:45045/ using document root <none> and password file <none>
I20260926 19:30:42.439055 16864 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:42.439203 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:42.439509 16864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:42.442391 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "83aad2e2796b4914a8b817e0729e172f"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "ff5bde401748fb2ca8588b15d222da77"
server_key_iv: "e26aa6605e9881d87f984c0a91ef2d99"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.443147 16864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "83aad2e2796b4914a8b817e0729e172f"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "ff5bde401748fb2ca8588b15d222da77"
server_key_iv: "e26aa6605e9881d87f984c0a91ef2d99"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.448387 16864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:30:42.451979 16879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:42.453477 16864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:30:42.453649 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "83aad2e2796b4914a8b817e0729e172f"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "ff5bde401748fb2ca8588b15d222da77"
server_key_iv: "e26aa6605e9881d87f984c0a91ef2d99"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.453862 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:42.466504 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:42.470474 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:42.470782 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:42.471709 16864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:42.473053 16864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:42.473160 16864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:42.473357 16864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:42.473446 16864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 19:30:42.510991 16864 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:32875
I20260926 19:30:42.511046 16991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:32875 every 8 connection(s)
I20260926 19:30:42.512244 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:30:42.515385 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16864
I20260926 19:30:42.515621 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:30:42.516160 1180 external_mini_cluster.cc:1428] Setting key d571f46a3d62d1068272a13ff808f05d
Sep 26 19:30:42 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:42.533394 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:42.514793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:38053 (local address 127.1.39.62:42859)
0926 19:30:42.514987 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:42.514991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:42.516113 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 19:30:42.518175 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:42.518179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:42.518274 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:42.518381 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:42.519835 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.521049 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.522604 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.523328 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.523474 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:42.527237 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:42.527246 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:42.527254 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:42.527302 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:42.530060 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:42.531051 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:42.531055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:42.531057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:42.531179 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:42.531676 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:42.531679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:42.531681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:42.531926 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:42.532038 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:42.532928 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 19:30:42.533095 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:30:42.534931 16994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:42.515125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:38053)
0926 19:30:42.515926 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:42.515975 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:42.517854 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:42.518580 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:42.518587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:42.519070 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:42.519670 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:42.519691 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.521185 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.521189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:42.522449 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:42.522461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:42.523524 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:42.523529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:42.523770 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:42.524531 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:42.524560 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:42.527001 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:42.530217 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:42.530228 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:42.530247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:42.530923 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:42.531294 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:42.531301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:42.531308 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:42.531563 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:42.532055 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:42.532066 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:42.532538 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 19:30:42.532842 (+ 304us) client_negotiation.cc:240] Negotiation successful
0926 19:30:42.533456 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":359,"threads_started":1}
I20260926 19:30:42.536617 16992 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:30:42.536995 16992 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:42.537833 16992 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
I20260926 19:30:42.539539 16669 ts_manager.cc:194] Registered new tserver with Master: 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875)
I20260926 19:30:42.540851 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:38053
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:30:42.577389 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:30:42.902245 16997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:30:42.902578 16997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:30:42.902683 16997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:30:42.909933 16997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:30:42.910054 16997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:30:42.910171 16997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:30:42.919832 16997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:30:42.921196 16997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:30:42.922603 16997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:30:42.936172 17002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:42.936374 17005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:30:42.936481 17003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:30:42.937757 16997 server_base.cc:1034] running on GCE node
I20260926 19:30:42.938342 16997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:30:42.939275 16997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:30:42.940469 16997 hybrid_clock.cc:648] HybridClock initialized: now 1790451042940432 us; error 31 us; skew 500 ppm
I20260926 19:30:42.940793 16997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:30:42 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:30:42.943938 16997 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:30:42.945459 16997 webserver.cc:460] Webserver started at http://127.1.39.3:34797/ using document root <none> and password file <none>
I20260926 19:30:42.946149 16997 fs_manager.cc:359] Metadata directory not provided
I20260926 19:30:42.946252 16997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:30:42.946589 16997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:30:42.949442 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "2ef6ab52a6ce4c7696b96131de776864"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "5d3583296052f59a6cbf746bdc32b68a"
server_key_iv: "74568d56b0250eab70795a8d6135c6b4"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.950201 16997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef6ab52a6ce4c7696b96131de776864"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "5d3583296052f59a6cbf746bdc32b68a"
server_key_iv: "74568d56b0250eab70795a8d6135c6b4"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.955132 16997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:30:42.958627 17012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:42.959983 16997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:30:42.960161 16997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "2ef6ab52a6ce4c7696b96131de776864"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "5d3583296052f59a6cbf746bdc32b68a"
server_key_iv: "74568d56b0250eab70795a8d6135c6b4"
server_key_version: "encryptionkey@0"
I20260926 19:30:42.960361 16997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:30:42.987914 16997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:30:42.991984 16997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:30:42.992290 16997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:30:42.993400 16997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:30:42.994745 16997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:30:42.994850 16997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:42.995008 16997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:30:42.995082 16997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:30:43.031145 16858 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:30:43.031478 16997 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37073
I20260926 19:30:43.031515 17124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37073 every 8 connection(s)
I20260926 19:30:43.032934 16997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:30:43.042284 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 16997
I20260926 19:30:43.042497 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:30:43.042961 1180 external_mini_cluster.cc:1428] Setting key 771fa9034a78dfb046955e41f6189ca0
Sep 26 19:30:43 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451042, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:30:43.052970 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.035810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:43911 (local address 127.1.39.62:42859)
0926 19:30:43.037667 (+ 1857us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.037671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.037688 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:30:43.039290 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.039294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.039365 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.039463 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:43.040877 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.041716 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.043347 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.043894 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.044029 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.047912 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:43.047916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:43.047920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:43.047951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:43.050548 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.051175 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.051179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.051181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.051291 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.051640 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.051642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.051644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.051880 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:43.051989 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.052567 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.052718 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1756,"thread_start_us":1649,"threads_started":1}
I20260926 19:30:43.053953 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.036424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:43911)
0926 19:30:43.037245 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.037313 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.039053 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.039654 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.039661 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.040153 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:30:43.040722 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.040740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.041848 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.041852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.043207 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.043219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.044128 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.044136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.044446 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.045195 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:30:43.045236 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:30:43.047718 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:30:43.050667 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:43.050680 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:43.050692 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:43.051078 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:43.051381 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:30:43.051385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:30:43.051387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:30:43.051552 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:30:43.051983 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:30:43.051988 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:30:43.052271 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.052489 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.052845 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":347,"threads_started":1}
I20260926 19:30:43.055228 17125 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:30:43.055473 17125 heartbeater.cc:464] Registering TS with master...
I20260926 19:30:43.056136 17125 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
I20260926 19:30:43.057705 16669 ts_manager.cc:194] Registered new tserver with Master: 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
I20260926 19:30:43.058696 16669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:43911
I20260926 19:30:43.058773 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:30:43.075804 17127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.063796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.1.39.62:42859)
0926 19:30:43.063984 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.063993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.064430 (+ 437us) server_negotiation.cc:407] Connection header received
0926 19:30:43.064540 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.064543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.064603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.064690 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:43.066118 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.066807 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.068061 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.068452 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.068545 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.070864 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:43.070868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:43.070872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:43.070899 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:43.073127 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.073876 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.073880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.073882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.074011 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.074494 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.074498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.074500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.074805 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:43.074989 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.075319 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.075473 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 19:30:43.079599 16669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:30:43.086362 16669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:30:43.108976 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.098655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.1.39.1:46457)
0926 19:30:43.099900 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.099909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.099937 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:30:43.100032 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.100040 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.100292 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.101374 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.101663 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.103272 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.106128 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.107784 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.107939 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.108185 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.108491 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.108654 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":926,"threads_started":1}
I20260926 19:30:43.109011 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.098219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:60304)
0926 19:30:43.099075 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.099124 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.099295 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.100545 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.100552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.100582 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.101121 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.101138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.103435 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.103444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.105948 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.105966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.108026 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.108042 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.108233 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.108285 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.108428 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.108610 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":8,"thread_start_us":1791,"threads_started":1}
I20260926 19:30:43.111397 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.098355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:37073 (local address 127.0.0.1:51612)
0926 19:30:43.100344 (+ 1989us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.100368 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.100474 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.103333 (+ 2859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.103345 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.103377 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.104323 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.104342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.105934 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.105939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.108128 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.108140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.110430 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.110437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.110634 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.110677 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.110751 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.110918 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"mutex_wait_us":329,"thread_start_us":1036,"threads_started":1}
I20260926 19:30:43.111804 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.100427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:32875 (local address 127.0.0.1:46450)
0926 19:30:43.101587 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.101611 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.101752 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.103010 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.103015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.103040 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.103842 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.103856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.105767 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.105774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.108497 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.108512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.110924 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.110931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.111144 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.111176 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.111259 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.111416 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":308,"thread_start_us":730,"threads_started":1}
I20260926 19:30:43.112654 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.099029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.1.39.3:37073)
0926 19:30:43.102727 (+ 3698us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.102735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.102766 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:30:43.102859 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.102867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.103106 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.103599 (+ 493us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.104539 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.105767 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.108290 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.110160 (+ 1870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.111770 (+ 1610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.112048 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.112124 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.112279 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3511,"thread_start_us":3356,"threads_started":1}
I20260926 19:30:43.112921 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.100981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46450 (local address 127.1.39.2:32875)
0926 19:30:43.102337 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.102348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.102382 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:30:43.102485 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.102492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.102784 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.103600 (+ 816us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.104092 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.105619 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.108702 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.110673 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.112162 (+ 1489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.112341 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.112413 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.112581 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1043,"threads_started":1}
I20260926 19:30:43.112673 16793 tablet_service.cc:1467] Processing CreateTablet for tablet bc9896bbd104476d98605309973db81d (DEFAULT_TABLE table=test-table [id=16905813b3c44d17ab8d0315d29373ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:43.114239 16793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9896bbd104476d98605309973db81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:43.116127 16927 tablet_service.cc:1467] Processing CreateTablet for tablet bc9896bbd104476d98605309973db81d (DEFAULT_TABLE table=test-table [id=16905813b3c44d17ab8d0315d29373ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:43.116245 17060 tablet_service.cc:1467] Processing CreateTablet for tablet bc9896bbd104476d98605309973db81d (DEFAULT_TABLE table=test-table [id=16905813b3c44d17ab8d0315d29373ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:30:43.117431 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9896bbd104476d98605309973db81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:43.117558 17060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9896bbd104476d98605309973db81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:30:43.131471 17146 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Bootstrap starting.
I20260926 19:30:43.131523 17144 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Bootstrap starting.
I20260926 19:30:43.134442 17146 tablet_bootstrap.cc:654] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:43.135685 17146 log.cc:824] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:43.136720 17144 tablet_bootstrap.cc:654] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:43.138501 17144 log.cc:824] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:43.138794 17146 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: No bootstrap required, opened a new log
I20260926 19:30:43.139120 17146 ts_tablet_manager.cc:1397] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260926 19:30:43.139948 17145 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Bootstrap starting.
I20260926 19:30:43.144022 17145 tablet_bootstrap.cc:654] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Neither blocks nor log segments found. Creating new log.
I20260926 19:30:43.144335 17144 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: No bootstrap required, opened a new log
I20260926 19:30:43.144670 17144 ts_tablet_manager.cc:1397] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 19:30:43.145411 17145 log.cc:824] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:30:43.147665 17146 raft_consensus.cc:348] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.148018 17146 raft_consensus.cc:374] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:43.148182 17146 raft_consensus.cc:729] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef6ab52a6ce4c7696b96131de776864, State: Initialized, Role: FOLLOWER
I20260926 19:30:43.148711 17145 tablet_bootstrap.cc:492] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: No bootstrap required, opened a new log
I20260926 19:30:43.148823 17146 consensus_queue.cc:260] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.149012 17145 ts_tablet_manager.cc:1397] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260926 19:30:43.154066 17125 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:30:43.154908 17146 ts_tablet_manager.cc:1428] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
I20260926 19:30:43.156850 17144 raft_consensus.cc:348] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.157363 17144 raft_consensus.cc:374] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:43.157543 17144 raft_consensus.cc:729] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5707388c209843ff9c56f27c103c75b0, State: Initialized, Role: FOLLOWER
I20260926 19:30:43.158289 17144 consensus_queue.cc:260] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.160079 17144 ts_tablet_manager.cc:1428] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260926 19:30:43.159845 17145 raft_consensus.cc:348] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.160329 17145 raft_consensus.cc:374] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:30:43.160465 17145 raft_consensus.cc:729] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aad2e2796b4914a8b817e0729e172f, State: Initialized, Role: FOLLOWER
I20260926 19:30:43.161218 17145 consensus_queue.cc:260] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.162622 16992 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:30:43.163188 17145 ts_tablet_manager.cc:1428] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260926 19:30:43.247529 16859 tablet.cc:2369] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:43.264329 16993 tablet.cc:2369] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:30:43.284878 17126 tablet.cc:2369] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:30:43.375804 17150 raft_consensus.cc:482] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:43.376118 17150 raft_consensus.cc:504] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.378094 17150 leader_election.cc:288] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457), 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875)
I20260926 19:30:43.383291 17152 raft_consensus.cc:482] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:30:43.383561 17152 raft_consensus.cc:504] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.385023 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.378480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.1.39.3:38757)
0926 19:30:43.378679 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.378697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.378822 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.379344 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.379349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.379381 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.379592 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.379602 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.381026 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.381032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.382608 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.382619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.384279 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.384285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.384469 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.384511 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.384605 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.384729 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260926 19:30:43.385566 17152 leader_election.cc:288] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
I20260926 19:30:43.385731 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.378704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:38757 (local address 127.1.39.1:46457)
0926 19:30:43.378889 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.378893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.378911 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:30:43.378972 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.378976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.379057 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.379163 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.379777 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.380889 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.382765 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.383940 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.384099 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.384194 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.385347 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.385474 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:30:43.386241 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.379187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:51335 (local address 127.1.39.2:32875)
0926 19:30:43.379383 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.379390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.379699 (+ 309us) server_negotiation.cc:407] Connection header received
0926 19:30:43.379886 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.379894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.379955 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.380064 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.380564 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.381469 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.383850 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.385123 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.385234 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.385347 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.385880 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.386006 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 19:30:43.386965 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.378858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:32875 (local address 127.1.39.3:51335)
0926 19:30:43.379598 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.379612 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.379746 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.380079 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.380083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.380105 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.380418 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.380426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.381623 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.381629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.383667 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.383690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.385510 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.385515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.385709 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.385741 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.386434 (+ 693us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.386562 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"spinlock_wait_cycles":25344,"thread_start_us":525,"threads_started":1}
I20260926 19:30:43.386683 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "2ef6ab52a6ce4c7696b96131de776864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5707388c209843ff9c56f27c103c75b0" is_pre_election: true
I20260926 19:30:43.387334 16813 raft_consensus.cc:2454] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef6ab52a6ce4c7696b96131de776864 in term 0.
I20260926 19:30:43.388546 17014 leader_election.cc:302] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef6ab52a6ce4c7696b96131de776864, 5707388c209843ff9c56f27c103c75b0; no voters:
I20260926 19:30:43.389706 16947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "2ef6ab52a6ce4c7696b96131de776864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aad2e2796b4914a8b817e0729e172f" is_pre_election: true
I20260926 19:30:43.390220 16947 raft_consensus.cc:2454] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef6ab52a6ce4c7696b96131de776864 in term 0.
I20260926 19:30:43.390810 17150 raft_consensus.cc:2781] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:30:43.390986 17150 raft_consensus.cc:482] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:30:43.391099 17150 raft_consensus.cc:3037] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:43.395057 17150 raft_consensus.cc:504] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.395854 17150 leader_election.cc:288] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [CANDIDATE]: Term 1 election: Requested vote from peers 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457), 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875)
I20260926 19:30:43.397214 16947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "2ef6ab52a6ce4c7696b96131de776864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aad2e2796b4914a8b817e0729e172f"
I20260926 19:30:43.397465 16947 raft_consensus.cc:3037] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:43.402201 16947 raft_consensus.cc:2454] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef6ab52a6ce4c7696b96131de776864 in term 1.
I20260926 19:30:43.402356 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "2ef6ab52a6ce4c7696b96131de776864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5707388c209843ff9c56f27c103c75b0"
I20260926 19:30:43.402652 17014 leader_election.cc:302] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f; no voters:
I20260926 19:30:43.402642 16813 raft_consensus.cc:3037] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:30:43.403059 17150 raft_consensus.cc:2781] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:30:43.406113 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.386743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:58897 (local address 127.1.39.1:46457)
0926 19:30:43.397754 (+ 11011us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.397761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.398350 (+ 589us) server_negotiation.cc:407] Connection header received
0926 19:30:43.398592 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.398597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.398661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.398714 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.399523 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.400626 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.403389 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.404797 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.404898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.405029 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.405676 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.405818 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10898,"thread_start_us":10716,"threads_started":1}
I20260926 19:30:43.406879 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.386084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.1.39.2:58897)
0926 19:30:43.398259 (+ 12175us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.398277 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.398456 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.398901 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.398905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.398934 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.399356 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.399367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.400805 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.400810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.403169 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.403196 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.405244 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.405250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.405475 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.405511 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.406337 (+ 826us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.406472 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"mutex_wait_us":4,"thread_start_us":15878,"threads_started":1}
I20260926 19:30:43.407284 17150 raft_consensus.cc:686] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 1 LEADER]: Becoming Leader. State: Replica: 2ef6ab52a6ce4c7696b96131de776864, State: Running, Role: LEADER
I20260926 19:30:43.407322 16813 raft_consensus.cc:2454] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef6ab52a6ce4c7696b96131de776864 in term 1.
I20260926 19:30:43.408370 17150 consensus_queue.cc:237] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } }
I20260926 19:30:43.408516 16813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "83aad2e2796b4914a8b817e0729e172f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5707388c209843ff9c56f27c103c75b0" is_pre_election: true
I20260926 19:30:43.408829 16813 raft_consensus.cc:2379] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83aad2e2796b4914a8b817e0729e172f in current term 1: Already voted for candidate 2ef6ab52a6ce4c7696b96131de776864 in this term.
I20260926 19:30:43.409469 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.387853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:36413 (local address 127.1.39.3:37073)
0926 19:30:43.388086 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.388094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.402371 (+ 14277us) server_negotiation.cc:407] Connection header received
0926 19:30:43.402598 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.402604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.402689 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.402752 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:30:43.403514 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.404588 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.407052 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.408283 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.408438 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.408537 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.409054 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.409175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:30:43.409832 17154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.387043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:37073 (local address 127.1.39.2:36413)
0926 19:30:43.402275 (+ 15232us) negotiation.cc:107] Waiting for socket to connect
0926 19:30:43.402294 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:30:43.402462 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:30:43.402917 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:30:43.402921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:30:43.402943 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:30:43.403353 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.403365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.404787 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.404791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.406861 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:30:43.406881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.408544 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.408551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:30:43.408875 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.408918 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 19:30:43.409029 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 19:30:43.409164 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15085,"mutex_wait_us":4,"thread_start_us":10862,"threads_started":1}
I20260926 19:30:43.410630 17080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9896bbd104476d98605309973db81d" candidate_uuid: "83aad2e2796b4914a8b817e0729e172f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef6ab52a6ce4c7696b96131de776864" is_pre_election: true
I20260926 19:30:43.411726 16883 leader_election.cc:302] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83aad2e2796b4914a8b817e0729e172f; no voters: 2ef6ab52a6ce4c7696b96131de776864, 5707388c209843ff9c56f27c103c75b0
I20260926 19:30:43.412191 17152 raft_consensus.cc:2726] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:30:43.413885 16669 catalog_manager.cc:5495] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "2ef6ab52a6ce4c7696b96131de776864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } health_report { overall_health: UNKNOWN } } }
I20260926 19:30:43.478076 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.472516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.1.39.3:37073)
0926 19:30:43.472727 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.472733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.472753 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:30:43.472964 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.472968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.473055 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.473233 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:30:43.473621 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.474365 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.475791 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.476197 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.476337 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.476863 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:30:43.477465 (+ 602us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:30:43.477618 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.477695 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.477773 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:30:43.486414 16947 raft_consensus.cc:1261] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Refusing update from remote peer 2ef6ab52a6ce4c7696b96131de776864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:43.487262 16813 raft_consensus.cc:1261] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Refusing update from remote peer 2ef6ab52a6ce4c7696b96131de776864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:30:43.488137 17157 consensus_queue.cc:1035] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:43.488621 17150 consensus_queue.cc:1035] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:30:43.516688 17161 mvcc.cc:205] Tried to move back new op lower bound from 7333687474108473344 to 7333687473806393344. Current Snapshot: MvccSnapshot[applied={T|T < 7333687474108473344}]
I20260926 19:30:43.528163 16669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:43.528471 16669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:30:43.535276 16669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Sending DeleteTablet for 3 replicas of tablet bc9896bbd104476d98605309973db81d
I20260926 19:30:43.536527 16793 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9896bbd104476d98605309973db81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:43 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:60304
I20260926 19:30:43.536947 17060 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9896bbd104476d98605309973db81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:43 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:51612
I20260926 19:30:43.537106 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16730
I20260926 19:30:43.545735 16927 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9896bbd104476d98605309973db81d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:30:43 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:46450
I20260926 19:30:43.554783 17171 tablet_replica.cc:330] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: stopping tablet replica
I20260926 19:30:43.556241 17171 raft_consensus.cc:2229] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:43.557585 17171 raft_consensus.cc:2258] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:43.560535 17171 ts_tablet_manager.cc:1905] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:43.567026 17174 tablet_replica.cc:330] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: stopping tablet replica
I20260926 19:30:43.567633 17174 raft_consensus.cc:2229] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:30:43.568037 17171 ts_tablet_manager.cc:1918] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:43.568207 17171 log.cc:1196] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/bc9896bbd104476d98605309973db81d
I20260926 19:30:43.568262 17174 raft_consensus.cc:2258] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:43.568737 17171 ts_tablet_manager.cc:1939] T bc9896bbd104476d98605309973db81d P 5707388c209843ff9c56f27c103c75b0: Deleting consensus metadata
I20260926 19:30:43.570807 17172 tablet_replica.cc:330] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: stopping tablet replica
I20260926 19:30:43.571208 16654 catalog_manager.cc:4841] TS 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457): tablet bc9896bbd104476d98605309973db81d (table test-table [id=16905813b3c44d17ab8d0315d29373ef]) successfully deleted
I20260926 19:30:43.571826 17172 raft_consensus.cc:2229] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:30:43.572276 17174 ts_tablet_manager.cc:1905] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:43.576336 17172 raft_consensus.cc:2258] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:30:43.578572 17172 ts_tablet_manager.cc:1905] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:30:43.579418 17174 ts_tablet_manager.cc:1918] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:43.579582 17174 log.cc:1196] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/bc9896bbd104476d98605309973db81d
I20260926 19:30:43.580093 17174 ts_tablet_manager.cc:1939] T bc9896bbd104476d98605309973db81d P 83aad2e2796b4914a8b817e0729e172f: Deleting consensus metadata
I20260926 19:30:43.581996 16654 catalog_manager.cc:4841] TS 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): tablet bc9896bbd104476d98605309973db81d (table test-table [id=16905813b3c44d17ab8d0315d29373ef]) successfully deleted
I20260926 19:30:43.584053 17172 ts_tablet_manager.cc:1918] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:30:43.584244 17172 log.cc:1196] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/bc9896bbd104476d98605309973db81d
I20260926 19:30:43.584707 17172 ts_tablet_manager.cc:1939] T bc9896bbd104476d98605309973db81d P 2ef6ab52a6ce4c7696b96131de776864: Deleting consensus metadata
I20260926 19:30:43.586225 16656 catalog_manager.cc:4841] TS 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): tablet bc9896bbd104476d98605309973db81d (table test-table [id=16905813b3c44d17ab8d0315d29373ef]) successfully deleted
Sep 26 19:30:43 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:30:43.593729 17140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.538967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.1.39.1:46457)
0926 19:30:43.539228 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.539238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.569423 (+ 30185us) server_negotiation.cc:407] Connection header received
0926 19:30:43.569751 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.569758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.569842 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.569919 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:43.572954 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.574181 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.576609 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.577258 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.577386 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.587846 (+ 10460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:43.587860 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:43.587874 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:43.587909 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:43.590785 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.591706 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.591713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.591724 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.591851 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.592271 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.592274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.592276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.592591 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:43.592805 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.593197 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.593449 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 19:30:43.594417 16853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:43.686004 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16730
I20260926 19:30:43.705366 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16864
Sep 26 19:30:43 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:30:43.722031 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.706154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.1.39.2:32875)
0926 19:30:43.706358 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.706364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.706389 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:30:43.706489 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.706495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.706578 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.706690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:43.707907 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.709045 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.710438 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.711067 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.711211 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.715451 (+ 4240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:43.715468 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:43.715488 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:43.715534 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:43.719162 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.719941 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.719951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.719955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.720118 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.720482 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.720487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.720490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.720923 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:43.721216 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.721437 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.721653 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 19:30:43.722780 16987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:43.814695 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16864
I20260926 19:30:43.833570 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16997
Sep 26 19:30:43 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:30:43.850841 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:30:43.834391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.1.39.3:37073)
0926 19:30:43.834575 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:30:43.834582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:30:43.834696 (+ 114us) server_negotiation.cc:407] Connection header received
0926 19:30:43.834915 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:30:43.834920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:30:43.835002 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:30:43.835120 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:30:43.836497 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.837385 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.838924 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:30:43.839506 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:30:43.839686 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:30:43.844660 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:30:43.844672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:30:43.844687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:30:43.844729 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:30:43.847829 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.848596 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.848603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.848609 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.848737 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:30:43.849142 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:30:43.849147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:30:43.849150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:30:43.849632 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:30:43.849952 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 19:30:43.850271 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:30:43.850511 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:30:43.851543 17120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:43.981803 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16997
I20260926 19:30:44.002948 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 16637
I20260926 19:30:44.004181 16699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:30:44.116951 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 16637
I20260926 19:31:04.140719 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--webserver_interface=127.1.39.62
--webserver_port=33857
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:04.473496 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:04.473863 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:04.474023 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:04.481418 17179 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:31:04.481566 17179 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:31:04.481674 17179 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:31:04.481761 17179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:04.481858 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:04.481979 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:04.482067 17179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:04.482146 17179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:04.494556 17179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=33857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:04.496136 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:04.497699 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:04.521488 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:04.521446 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:04.521518 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:04.522497 17179 server_base.cc:1034] running on GCE node
I20260926 19:31:04.523372 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:04.524933 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:04.526185 17179 hybrid_clock.cc:648] HybridClock initialized: now 1790451064526083 us; error 79 us; skew 500 ppm
I20260926 19:31:04.526582 17179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:04 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451064, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:04.535142 17179 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:31:04.536923 17179 webserver.cc:460] Webserver started at http://127.1.39.62:33857/ using document root <none> and password file <none>
I20260926 19:31:04.537801 17179 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:04.537951 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:04.544003 17179 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 19:31:04.547335 17195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:04.549320 17179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:31:04.549551 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "a0f54fb07dcf400b8066ec39f99fa531"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0c28d91eff7117bcdb0035e3e8aa1364"
server_key_iv: "c9428010b230fc597f7effaa18706e0e"
server_key_version: "encryptionkey@0"
I20260926 19:31:04.550987 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:04.571743 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:04.576246 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:04.576526 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:04.600073 17179 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42859
I20260926 19:31:04.600106 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42859 every 8 connection(s)
I20260926 19:31:04.601778 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:04.606967 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17179
I20260926 19:31:04.607822 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:46457
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=43967
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:31:04.609246 17247 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:31:04.612560 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap starting.
I20260926 19:31:04.640676 17247 log.cc:824] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:04.656376 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:31:04.657083 17247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap complete.
I20260926 19:31:04.670921 17247 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:04.672155 17247 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Initialized, Role: FOLLOWER
I20260926 19:31:04.672884 17247 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:04.673174 17247 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:04.673321 17247 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:04.673497 17247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 1 FOLLOWER]: Advancing to term 2
I20260926 19:31:04.680049 17247 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:04.680748 17247 leader_election.cc:302] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0f54fb07dcf400b8066ec39f99fa531; no voters:
I20260926 19:31:04.681525 17247 leader_election.cc:288] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 19:31:04.681686 17251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 19:31:04.683074 17251 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 LEADER]: Becoming Leader. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Running, Role: LEADER
I20260926 19:31:04.683764 17247 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:04.683691 17251 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:04.686084 17253 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f54fb07dcf400b8066ec39f99fa531. Latest consensus state: current_term: 2 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:31:04.686441 17253 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:04.686966 17252 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:31:04.687284 17252 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:04.697665 17258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:04.706110 17258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:04.707584 17258 catalog_manager.cc:1259] Loaded cluster ID: 3baba09097db4cf591317ac2ce6ebca1
I20260926 19:31:04.707753 17258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:04.711820 17258 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:04.715394 17258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Loaded TSK: 0
I20260926 19:31:04.733333 17268 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 19:31:04.750341 17258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Generated new TSK 1
I20260926 19:31:04.750779 17258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 19:31:04.993781 17249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:04.994102 17249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:04.994267 17249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:05.002000 17249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:05.002156 17249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:05.002331 17249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:05.012161 17249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=43967
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:05.013523 17249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:05.014842 17249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:05.027784 17274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.027899 17275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.028239 17277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:05.029397 17249 server_base.cc:1034] running on GCE node
I20260926 19:31:05.030092 17249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:05.031208 17249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:05.032437 17249 hybrid_clock.cc:648] HybridClock initialized: now 1790451065032397 us; error 36 us; skew 500 ppm
I20260926 19:31:05.032768 17249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:05 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:05.036068 17249 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:31:05.037971 17249 webserver.cc:460] Webserver started at http://127.1.39.1:43967/ using document root <none> and password file <none>
I20260926 19:31:05.038686 17249 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:05.038825 17249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:05.044294 17249 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 19:31:05.047176 17284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.048619 17249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:31:05.048780 17249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "5707388c209843ff9c56f27c103c75b0"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0af587ffad56a458139bd8b0abb79418"
server_key_iv: "d65699405e4383ddcbaaf6329bd32dff"
server_key_version: "encryptionkey@0"
I20260926 19:31:05.049815 17249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:05.082908 17249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:05.087069 17249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:05.087337 17249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:05.088406 17249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:05.090209 17249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:05.090374 17249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 19:31:05.090508 17249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:05.090605 17249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.125538 17249 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:46457
I20260926 19:31:05.125633 17396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:46457 every 8 connection(s)
I20260926 19:31:05.126958 17249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:05.129357 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17249
I20260926 19:31:05.129993 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:32875
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=45045
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:31:05 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:05.151641 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:05.130587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:45393 (local address 127.1.39.62:42859)
0926 19:31:05.133747 (+ 3160us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:05.133758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:05.133797 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:31:05.134088 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:05.134101 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:05.134414 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:05.134771 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:05.136294 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.137577 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.139165 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.139816 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.140014 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:05.143842 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:05.143850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:05.143865 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:05.143905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:05.146999 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:05.147787 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:05.147800 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:05.147805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:05.147968 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:05.148410 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:05.148416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:05.148419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:05.148890 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:05.149098 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:05.149966 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 19:31:05.150361 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2974,"thread_start_us":2835,"threads_started":1}
I20260926 19:31:05.152140 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:05.130083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:45393)
0926 19:31:05.130979 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:05.131031 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:05.132747 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:05.134922 (+ 2175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:05.134933 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:05.135559 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:05.136088 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:05.136106 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.137788 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.137794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:05.139003 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:05.139014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.140060 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.140068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:05.140331 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:05.141232 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:05.141255 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:05.143546 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:05.147168 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:05.147177 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:05.147193 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:05.147633 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:05.148099 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:05.148104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:05.148107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:05.148295 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:05.149040 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:05.149047 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:05.149484 (+ 437us) client_negotiation.cc:769] Sending connection context
0926 19:31:05.149769 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 19:31:05.150329 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":369,"threads_started":1}
I20260926 19:31:05.155244 17397 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:31:05.155596 17397 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:05.156595 17397 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
I20260926 19:31:05.158965 17212 ts_manager.cc:194] Registered new tserver with Master: 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457)
I20260926 19:31:05.160503 17212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:45393
W20260926 19:31:05.468775 17401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:05.469085 17401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:05.469262 17401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:05.477082 17401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:05.477233 17401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:05.477437 17401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:05.487205 17401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:32875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=45045
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:05.488463 17401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:05.489790 17401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:05.503021 17406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.502929 17409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.502969 17407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:05.503877 17401 server_base.cc:1034] running on GCE node
I20260926 19:31:05.504484 17401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:05.505409 17401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:05.506649 17401 hybrid_clock.cc:648] HybridClock initialized: now 1790451065506594 us; error 39 us; skew 500 ppm
I20260926 19:31:05.506970 17401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:05 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:05.510084 17401 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:31:05.511555 17401 webserver.cc:460] Webserver started at http://127.1.39.2:45045/ using document root <none> and password file <none>
I20260926 19:31:05.512214 17401 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:05.512370 17401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:05.517694 17401 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:05.520375 17416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.521768 17401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:05.521936 17401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "83aad2e2796b4914a8b817e0729e172f"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "ff5bde401748fb2ca8588b15d222da77"
server_key_iv: "e26aa6605e9881d87f984c0a91ef2d99"
server_key_version: "encryptionkey@0"
I20260926 19:31:05.522899 17401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:05.540030 17401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:05.544236 17401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:05.544513 17401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:05.545678 17401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:05.546983 17401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:05.547103 17401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.547242 17401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:05.547330 17401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.580781 17401 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:32875
I20260926 19:31:05.580897 17528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:32875 every 8 connection(s)
I20260926 19:31:05.582340 17401 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:05.586692 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17401
I20260926 19:31:05.587307 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:37073
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=34797
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 19:31:05 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:05.602485 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:05.584903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:55881 (local address 127.1.39.62:42859)
0926 19:31:05.585070 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:05.585074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:05.586230 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 19:31:05.588187 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:05.588193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:05.588285 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:05.588398 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:05.589912 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.590616 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.592138 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.592648 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.592791 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:05.596168 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:05.596172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:05.596175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:05.596207 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:05.598815 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:05.599659 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:05.599667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:05.599674 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:05.599836 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:05.600346 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:05.600354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:05.600360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:05.600783 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:05.600935 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:05.601760 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 19:31:05.602031 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:31:05.603439 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:05.585394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:55881)
0926 19:31:05.586070 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:05.586111 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:05.587915 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:05.588625 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:05.588633 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:05.589118 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:05.589743 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:05.589760 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.590781 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.590787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:05.591983 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:05.591995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:05.592883 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:05.592890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:05.593118 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:05.593840 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:05.593862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:05.595938 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:05.598956 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:05.598965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:05.598979 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:05.599503 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:05.599960 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:05.599964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:05.599968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:05.600200 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:05.600926 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:05.600934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:05.601321 (+ 387us) client_negotiation.cc:769] Sending connection context
0926 19:31:05.601613 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 19:31:05.602111 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260926 19:31:05.604985 17529 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:31:05.605332 17529 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:05.606184 17529 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
I20260926 19:31:05.607908 17212 ts_manager.cc:194] Registered new tserver with Master: 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875)
I20260926 19:31:05.608963 17212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:55881
W20260926 19:31:05.917811 17532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:05.918139 17532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:05.918306 17532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:05.925772 17532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:05.925928 17532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:05.926096 17532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:05.936282 17532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=34797
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:05.937779 17532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:05.939102 17532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:05.952323 17540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.952391 17537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:05.952355 17538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:05.953001 17532 server_base.cc:1034] running on GCE node
I20260926 19:31:05.953830 17532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:05.954981 17532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:05.956224 17532 hybrid_clock.cc:648] HybridClock initialized: now 1790451065956167 us; error 61 us; skew 500 ppm
I20260926 19:31:05.956555 17532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:05 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:05.959669 17532 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:31:05.961174 17532 webserver.cc:460] Webserver started at http://127.1.39.3:34797/ using document root <none> and password file <none>
I20260926 19:31:05.961871 17532 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:05.961984 17532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:05.967422 17532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 19:31:05.970171 17547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.971498 17532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:31:05.971668 17532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "2ef6ab52a6ce4c7696b96131de776864"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "5d3583296052f59a6cbf746bdc32b68a"
server_key_iv: "74568d56b0250eab70795a8d6135c6b4"
server_key_version: "encryptionkey@0"
I20260926 19:31:05.972581 17532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:05.987525 17532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:05.991535 17532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:05.991818 17532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:05.992849 17532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:05.994302 17532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:05.994386 17532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:05.994539 17532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:05.994617 17532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:06.028540 17532 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37073
I20260926 19:31:06.028623 17659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37073 every 8 connection(s)
I20260926 19:31:06.030092 17532 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:06.033744 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17532
Sep 26 19:31:06 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451065, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:06.049952 17664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.034764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.1.39.62:42859)
0926 19:31:06.035834 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.035841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.035858 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:31:06.035919 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.035924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.036005 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.036130 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.037906 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.038833 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.040336 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.040972 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.041124 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.043947 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.043951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.043954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.043982 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.047097 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.047797 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.047803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.047816 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.047933 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.048324 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.048329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.048333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.048723 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.048990 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.049271 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.049483 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":864,"threads_started":1}
I20260926 19:31:06.051894 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.032551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:60709 (local address 127.1.39.62:42859)
0926 19:31:06.032731 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.032736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.033950 (+ 1214us) server_negotiation.cc:407] Connection header received
0926 19:31:06.036514 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.036520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.036597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.036735 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.038377 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.039284 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.041115 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.041807 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.041932 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.045466 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.045470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.045473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.045511 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.048401 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.049323 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.049328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.049331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.049473 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.049952 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.049956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.049958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.050324 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.050491 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.051359 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.051539 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:31:06.053021 17662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.032904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:60709)
0926 19:31:06.033774 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.033825 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.036250 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.036908 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.036916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.037482 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:06.038204 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.038228 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.039438 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.039441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.040958 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.040970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.042028 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.042037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.042250 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.042853 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:06.042874 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:06.045243 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:06.048547 (+ 3304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:06.048555 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:06.048572 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:06.049167 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:06.049603 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:06.049608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:06.049612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:06.049844 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:06.050469 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:06.050475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:06.050875 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.051232 (+ 357us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.051734 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":351,"threads_started":1}
I20260926 19:31:06.054384 17660 heartbeater.cc:347] Connected to a master server at 127.1.39.62:42859
I20260926 19:31:06.054728 17660 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:06.055577 17660 heartbeater.cc:511] Master 127.1.39.62:42859 requested a full tablet report, sending...
I20260926 19:31:06.056937 17212 ts_manager.cc:194] Registered new tserver with Master: 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
I20260926 19:31:06.057912 17212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:60709
I20260926 19:31:06.065696 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:31:06.072192 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.066688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.1.39.62:42859)
0926 19:31:06.066926 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.066936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.067302 (+ 366us) server_negotiation.cc:407] Connection header received
0926 19:31:06.067530 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.067534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.067687 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.067826 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:06.068233 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.068972 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.070482 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.070883 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.070980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.071427 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:06.071651 (+ 224us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 19:31:06.071938 (+ 287us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35860: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 19:31:06.072332 17400 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35860: token expired
W20260926 19:31:06.072816 17130 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.39.62:42859)
I20260926 19:31:06.085808 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.073374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.1.39.62:42859)
0926 19:31:06.073527 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.073531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.073546 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:31:06.073583 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.073586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.073651 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.073728 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.075045 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.075787 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.077209 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.077664 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.077761 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.080219 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.080223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.080226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.080255 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.082844 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.083426 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.083430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.083432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.083588 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.084067 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.084070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.084071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.084416 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.084698 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.085189 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.085454 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 19:31:06.089205 17212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:31:06.097966 17212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:31:06.124045 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.111192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.1.39.3:37073)
0926 19:31:06.113900 (+ 2708us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.113911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.114305 (+ 394us) server_negotiation.cc:407] Connection header received
0926 19:31:06.114598 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.114605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.114778 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.114959 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:06.115930 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.117623 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.121147 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.122658 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.122779 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.123004 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.123525 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.123687 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2521,"thread_start_us":2351,"threads_started":1}
I20260926 19:31:06.124543 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.109622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.1.39.1:46457)
0926 19:31:06.114347 (+ 4725us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.114355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.114379 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:31:06.114483 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.114491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.114779 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.115805 (+ 1026us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:06.116073 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.117634 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.121184 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.122472 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.123909 (+ 1437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.124102 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.124166 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.124315 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4563,"thread_start_us":4450,"threads_started":1}
I20260926 19:31:06.124842 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.110430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:37073 (local address 127.0.0.1:52600)
0926 19:31:06.114182 (+ 3752us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.114206 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.114417 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.115319 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.115323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.115344 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:06.115743 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.115754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.117810 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.117814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.120955 (+ 3141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.120978 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.123155 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.123167 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.123325 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.123351 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.123459 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.123667 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3619,"mutex_wait_us":74,"thread_start_us":999,"threads_started":1}
I20260926 19:31:06.125193 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.109474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39620)
0926 19:31:06.111281 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.111334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.111539 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.115029 (+ 3490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.115038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.115076 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:06.115562 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.115574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.117795 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.117802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.120988 (+ 3186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.120997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.122718 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.122726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.122913 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.122970 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.123106 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.123341 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1527,"mutex_wait_us":55,"thread_start_us":4428,"threads_started":1}
I20260926 19:31:06.128981 17674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.114564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:32875 (local address 127.0.0.1:38720)
0926 19:31:06.118712 (+ 4148us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.118734 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.118839 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.120698 (+ 1859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.120703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.120728 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:06.121234 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.121246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.123125 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.123134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.126212 (+ 3078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.126231 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.128265 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.128271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.128440 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.128470 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.128561 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.128683 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3996,"thread_start_us":3828,"threads_started":1}
I20260926 19:31:06.129012 17332 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 (DEFAULT_TABLE table=test-table [id=801bd977eb3e405aa9212e4cb5a7e444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:06.128999 17595 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 (DEFAULT_TABLE table=test-table [id=801bd977eb3e405aa9212e4cb5a7e444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:06.129743 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.114586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.1.39.2:32875)
0926 19:31:06.120199 (+ 5613us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.120207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.120241 (+ 34us) server_negotiation.cc:407] Connection header received
0926 19:31:06.120320 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.120325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.120505 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.120715 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:06.121457 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.122975 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.126405 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.127831 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.127954 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.128129 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.129223 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.129404 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5479,"thread_start_us":5377,"threads_started":1}
I20260926 19:31:06.130342 17595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebe5c3d008e4117a309e690e8cae686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:06.130411 17332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebe5c3d008e4117a309e690e8cae686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:06.137928 17464 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 (DEFAULT_TABLE table=test-table [id=801bd977eb3e405aa9212e4cb5a7e444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:06.139305 17464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebe5c3d008e4117a309e690e8cae686. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:06.143750 17676 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Bootstrap starting.
I20260926 19:31:06.145164 17677 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Bootstrap starting.
I20260926 19:31:06.147462 17676 tablet_bootstrap.cc:654] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:06.148749 17676 log.cc:824] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:06.149006 17677 tablet_bootstrap.cc:654] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:06.150228 17677 log.cc:824] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:06.152209 17676 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: No bootstrap required, opened a new log
I20260926 19:31:06.152485 17676 ts_tablet_manager.cc:1397] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260926 19:31:06.153156 17677 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: No bootstrap required, opened a new log
I20260926 19:31:06.153479 17677 ts_tablet_manager.cc:1397] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260926 19:31:06.154258 17680 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Bootstrap starting.
I20260926 19:31:06.157743 17680 tablet_bootstrap.cc:654] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:06.158730 17680 log.cc:824] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:06.161552 17680 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: No bootstrap required, opened a new log
I20260926 19:31:06.161808 17680 ts_tablet_manager.cc:1397] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260926 19:31:06.164099 17397 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:31:06.163787 17676 raft_consensus.cc:348] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.164062 17677 raft_consensus.cc:348] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.164445 17677 raft_consensus.cc:374] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:06.164489 17676 raft_consensus.cc:374] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:06.164561 17677 raft_consensus.cc:729] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef6ab52a6ce4c7696b96131de776864, State: Initialized, Role: FOLLOWER
I20260926 19:31:06.164728 17676 raft_consensus.cc:729] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5707388c209843ff9c56f27c103c75b0, State: Initialized, Role: FOLLOWER
I20260926 19:31:06.165206 17677 consensus_queue.cc:260] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.165326 17676 consensus_queue.cc:260] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.171739 17660 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:31:06.171510 17680 raft_consensus.cc:348] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.171978 17680 raft_consensus.cc:374] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:06.172094 17680 raft_consensus.cc:729] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83aad2e2796b4914a8b817e0729e172f, State: Initialized, Role: FOLLOWER
I20260926 19:31:06.172711 17677 ts_tablet_manager.cc:1428] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260926 19:31:06.172817 17680 consensus_queue.cc:260] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.174149 17529 heartbeater.cc:503] Master 127.1.39.62:42859 was elected leader, sending a full tablet report...
I20260926 19:31:06.175208 17676 ts_tablet_manager.cc:1428] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Time spent starting tablet: real 0.023s user 0.009s sys 0.010s
I20260926 19:31:06.176127 17680 ts_tablet_manager.cc:1428] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260926 19:31:06.281989 17661 tablet.cc:2369] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:06.334406 17530 tablet.cc:2369] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:31:06.352030 17683 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:06.352305 17683 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.354096 17683 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
I20260926 19:31:06.360975 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.354962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:60023 (local address 127.1.39.2:32875)
0926 19:31:06.355186 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.355192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.355406 (+ 214us) server_negotiation.cc:407] Connection header received
0926 19:31:06.355666 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.355670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.355731 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.355828 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:06.356359 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.357149 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.359223 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.360048 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.360208 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.360316 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.360594 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.360720 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:31:06.361188 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.354485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:32875 (local address 127.1.39.1:60023)
0926 19:31:06.355286 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.355308 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.355494 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.355919 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.355924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.355960 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:06.356220 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.356227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.357321 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.357329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.359068 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.359080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.360282 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.360288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.360437 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.360461 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.360568 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.360695 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":1,"thread_start_us":439,"threads_started":1}
I20260926 19:31:06.362104 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.354650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:59245 (local address 127.1.39.3:37073)
0926 19:31:06.355207 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.355216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.355851 (+ 635us) server_negotiation.cc:407] Connection header received
0926 19:31:06.356096 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.356109 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.356174 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.356282 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:06.356860 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.357818 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.359835 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.360766 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.360921 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.361057 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.361677 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.361823 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391}
I20260926 19:31:06.362286 17484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebe5c3d008e4117a309e690e8cae686" candidate_uuid: "5707388c209843ff9c56f27c103c75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aad2e2796b4914a8b817e0729e172f" is_pre_election: true
I20260926 19:31:06.362788 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.354608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:37073 (local address 127.1.39.1:59245)
0926 19:31:06.355740 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.355756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:06.355921 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:06.356296 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:06.356301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:06.356326 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:06.356701 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.356710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.358028 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.358034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.359671 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:06.359684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.361101 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.361110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:06.361382 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.361426 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 19:31:06.362303 (+ 877us) client_negotiation.cc:240] Negotiation successful
0926 19:31:06.362439 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":827,"threads_started":1}
I20260926 19:31:06.362838 17484 raft_consensus.cc:2454] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5707388c209843ff9c56f27c103c75b0 in term 0.
I20260926 19:31:06.363515 17286 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0, 83aad2e2796b4914a8b817e0729e172f; no voters:
I20260926 19:31:06.363891 17683 raft_consensus.cc:2781] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:31:06.364015 17683 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:31:06.364135 17683 raft_consensus.cc:3037] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:06.363919 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebe5c3d008e4117a309e690e8cae686" candidate_uuid: "5707388c209843ff9c56f27c103c75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef6ab52a6ce4c7696b96131de776864" is_pre_election: true
I20260926 19:31:06.364486 17615 raft_consensus.cc:2454] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5707388c209843ff9c56f27c103c75b0 in term 0.
I20260926 19:31:06.366459 17683 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.367086 17683 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 1 election: Requested vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
I20260926 19:31:06.367475 17484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebe5c3d008e4117a309e690e8cae686" candidate_uuid: "5707388c209843ff9c56f27c103c75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83aad2e2796b4914a8b817e0729e172f"
I20260926 19:31:06.367684 17484 raft_consensus.cc:3037] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:06.367664 17615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebe5c3d008e4117a309e690e8cae686" candidate_uuid: "5707388c209843ff9c56f27c103c75b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef6ab52a6ce4c7696b96131de776864"
I20260926 19:31:06.367854 17615 raft_consensus.cc:3037] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:06.370976 17615 raft_consensus.cc:2454] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5707388c209843ff9c56f27c103c75b0 in term 1.
I20260926 19:31:06.370976 17484 raft_consensus.cc:2454] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5707388c209843ff9c56f27c103c75b0 in term 1.
I20260926 19:31:06.371383 17286 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2ef6ab52a6ce4c7696b96131de776864, 5707388c209843ff9c56f27c103c75b0, 83aad2e2796b4914a8b817e0729e172f; no voters:
I20260926 19:31:06.371706 17683 raft_consensus.cc:2781] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:06.372161 17683 raft_consensus.cc:686] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 LEADER]: Becoming Leader. State: Replica: 5707388c209843ff9c56f27c103c75b0, State: Running, Role: LEADER
I20260926 19:31:06.372618 17683 consensus_queue.cc:237] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:06.376673 17211 catalog_manager.cc:5495] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5707388c209843ff9c56f27c103c75b0 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "5707388c209843ff9c56f27c103c75b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } health_report { overall_health: UNKNOWN } } }
W20260926 19:31:06.383720 17398 tablet.cc:2369] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:31:06.432807 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.427653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39634 (local address 127.1.39.1:46457)
0926 19:31:06.427840 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.427845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.427880 (+ 35us) server_negotiation.cc:407] Connection header received
0926 19:31:06.427993 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.427996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.428078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.428198 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:06.428603 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.429416 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.430866 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.431284 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.431381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.431812 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:06.432310 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:31:06.432416 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.432521 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.432588 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:31:06.441056 17484 raft_consensus.cc:1261] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Refusing update from remote peer 5707388c209843ff9c56f27c103c75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:06.441056 17615 raft_consensus.cc:1261] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Refusing update from remote peer 5707388c209843ff9c56f27c103c75b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:06.442073 17687 consensus_queue.cc:1035] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:06.442404 17683 consensus_queue.cc:1035] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:06.453521 17690 mvcc.cc:205] Tried to move back new op lower bound from 7333687568132579328 to 7333687567865962496. Current Snapshot: MvccSnapshot[applied={T|T < 7333687568132579328}]
I20260926 19:31:06.462317 17692 mvcc.cc:205] Tried to move back new op lower bound from 7333687568132579328 to 7333687567865962496. Current Snapshot: MvccSnapshot[applied={T|T < 7333687568132579328}]
I20260926 19:31:06.468820 17212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:06.469130 17212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:06.472609 17212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Sending DeleteTablet for 3 replicas of tablet 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:06.473835 17464 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:38720
I20260926 19:31:06.473835 17595 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:52600
I20260926 19:31:06.473919 17332 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ebe5c3d008e4117a309e690e8cae686 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:06 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:39620
I20260926 19:31:06.474962 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17249
I20260926 19:31:06.514245 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.476928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.1.39.1:46457)
0926 19:31:06.477125 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.477134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.477157 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:31:06.477232 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.477240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.477364 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.477497 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.480059 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.481753 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.491482 (+ 9729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.493055 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.493230 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.501180 (+ 7950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.501204 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.501236 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.501326 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.506908 (+ 5582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.508192 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.508206 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.508213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.508448 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.509693 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.509701 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.509707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.510341 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.510792 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.513269 (+ 2477us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.513631 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 19:31:06.515466 17392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:06.519742 17700 tablet_replica.cc:330] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: stopping tablet replica
I20260926 19:31:06.520179 17700 raft_consensus.cc:2229] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:06.520674 17700 raft_consensus.cc:2258] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:06.523432 17700 ts_tablet_manager.cc:1905] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:06.524505 17701 tablet_replica.cc:330] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: stopping tablet replica
I20260926 19:31:06.524958 17701 raft_consensus.cc:2229] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:06.525514 17701 raft_consensus.cc:2258] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:06.528187 17701 ts_tablet_manager.cc:1905] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:06.531406 17700 ts_tablet_manager.cc:1918] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:06.531580 17700 log.cc:1196] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:06.532152 17700 ts_tablet_manager.cc:1939] T 1ebe5c3d008e4117a309e690e8cae686 P 83aad2e2796b4914a8b817e0729e172f: Deleting consensus metadata
I20260926 19:31:06.534248 17197 catalog_manager.cc:4841] TS 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): tablet 1ebe5c3d008e4117a309e690e8cae686 (table test-table [id=801bd977eb3e405aa9212e4cb5a7e444]) successfully deleted
I20260926 19:31:06.535313 17701 ts_tablet_manager.cc:1918] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:06.535475 17701 log.cc:1196] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:06.535924 17701 ts_tablet_manager.cc:1939] T 1ebe5c3d008e4117a309e690e8cae686 P 2ef6ab52a6ce4c7696b96131de776864: Deleting consensus metadata
I20260926 19:31:06.537492 17199 catalog_manager.cc:4841] TS 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): tablet 1ebe5c3d008e4117a309e690e8cae686 (table test-table [id=801bd977eb3e405aa9212e4cb5a7e444]) successfully deleted
I20260926 19:31:06.626273 17702 tablet_replica.cc:330] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: stopping tablet replica
I20260926 19:31:06.626646 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17249
W20260926 19:31:06.649816 17197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.39.1:46457 (error 108)
I20260926 19:31:06.650699 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17401
I20260926 19:31:06.652045 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.651137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39654)
0926 19:31:06.651672 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.651821 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
W20260926 19:31:06.652371 17197 catalog_manager.cc:4584] TS 5707388c209843ff9c56f27c103c75b0 (127.1.39.1:46457): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ebe5c3d008e4117a309e690e8cae686: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
I20260926 19:31:06.669274 17675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.651437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38730 (local address 127.1.39.2:32875)
0926 19:31:06.651629 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.651635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.651786 (+ 151us) server_negotiation.cc:407] Connection header received
0926 19:31:06.651945 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.651951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.652053 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.652175 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.653698 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.654455 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.655987 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.656383 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.656510 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.663577 (+ 7067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.663591 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.663607 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.663642 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.666619 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.667346 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.667353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.667355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.667481 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.667856 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.667859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.667861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.668230 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.668442 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.668734 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.668896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:31:06.670037 17524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:06.706264 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.705760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39656)
0926 19:31:06.706008 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.706119 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 19:31:06.707360 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.706967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39658)
0926 19:31:06.707165 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.707232 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:31:06.761438 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17401
I20260926 19:31:06.767989 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.767520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39664)
0926 19:31:06.767743 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.767829 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 19:31:06.769172 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.768765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39668)
0926 19:31:06.768960 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.769028 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 19:31:06.779102 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17532
I20260926 19:31:06.792538 17673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.779904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.1.39.3:37073)
0926 19:31:06.780090 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:06.780095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:06.780112 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:31:06.780241 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:06.780246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:06.780311 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:06.780393 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:06.781686 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.782390 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.783679 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:06.784112 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:06.784248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:06.786969 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:06.786982 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:06.786998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:06.787034 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:06.789894 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.790629 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.790635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.790638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.790800 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:06.791172 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:06.791175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:06.791177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:06.791501 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:06.791730 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:06.792000 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 19:31:06.792155 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:31:06.793253 17655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:06.850139 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.849645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39682)
0926 19:31:06.849896 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.849995 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 19:31:06.851336 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:06.850924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46457 (local address 127.0.0.1:39694)
0926 19:31:06.851125 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:06.851202 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 19:31:06.882530 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17532
I20260926 19:31:06.900803 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17179
I20260926 19:31:06.902108 17242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:07.019250 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17179
I20260926 19:31:27.042935 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--webserver_interface=127.1.39.62
--webserver_port=33857
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:27.375051 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:27.375351 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:27.375442 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:27.382555 17709 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 19:31:27.382658 17709 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 19:31:27.382696 17709 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 19:31:27.382741 17709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:27.382795 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:27.382858 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:27.382905 17709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:27.382951 17709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:27.392673 17709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=33857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:27.393877 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:27.395098 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:27.406754 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:27.406810 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:27.406831 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:27.407789 17709 server_base.cc:1034] running on GCE node
I20260926 19:31:27.408566 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:27.409997 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:27.411258 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790451087411185 us; error 28 us; skew 500 ppm
I20260926 19:31:27.411687 17709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:27 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:27.420200 17709 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:31:27.421764 17709 webserver.cc:460] Webserver started at http://127.1.39.62:33857/ using document root <none> and password file <none>
I20260926 19:31:27.422479 17709 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:27.422601 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:27.428054 17709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 19:31:27.430841 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:27.432066 17709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:31:27.432251 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "a0f54fb07dcf400b8066ec39f99fa531"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0c28d91eff7117bcdb0035e3e8aa1364"
server_key_iv: "c9428010b230fc597f7effaa18706e0e"
server_key_version: "encryptionkey@0"
I20260926 19:31:27.433205 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:27.449492 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:27.453477 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:27.453752 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:27.472427 17709 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42859
I20260926 19:31:27.472525 17775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42859 every 8 connection(s)
I20260926 19:31:27.473917 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:27.478312 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17709
I20260926 19:31:27.479082 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:46457
--local_ip_for_outbound_sockets=127.1.39.1
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=43967
--webserver_interface=127.1.39.1
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:31:27.480816 17776 sys_catalog.cc:263] Verifying existing consensus state
I20260926 19:31:27.484208 17776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap starting.
I20260926 19:31:27.504318 17776 log.cc:824] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:27.520799 17776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:31:27.521481 17776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Bootstrap complete.
I20260926 19:31:27.530853 17776 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:27.531807 17776 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Initialized, Role: FOLLOWER
I20260926 19:31:27.532500 17776 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:27.532750 17776 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:27.532879 17776 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:27.533066 17776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 2 FOLLOWER]: Advancing to term 3
I20260926 19:31:27.537269 17776 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:27.537850 17776 leader_election.cc:302] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0f54fb07dcf400b8066ec39f99fa531; no voters:
I20260926 19:31:27.538622 17776 leader_election.cc:288] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 19:31:27.538772 17780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 19:31:27.539840 17780 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [term 3 LEADER]: Becoming Leader. State: Replica: a0f54fb07dcf400b8066ec39f99fa531, State: Running, Role: LEADER
I20260926 19:31:27.540406 17780 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } }
I20260926 19:31:27.540848 17776 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:27.543829 17782 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0f54fb07dcf400b8066ec39f99fa531. Latest consensus state: current_term: 3 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:31:27.544689 17782 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:27.544219 17781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a0f54fb07dcf400b8066ec39f99fa531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0f54fb07dcf400b8066ec39f99fa531" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42859 } } }
I20260926 19:31:27.545223 17781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:27.556581 17785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:27.565323 17785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:27.566761 17785 catalog_manager.cc:1259] Loaded cluster ID: 3baba09097db4cf591317ac2ce6ebca1
I20260926 19:31:27.566871 17785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:27.570914 17785 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:27.572530 17785 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Loaded TSK: 0
I20260926 19:31:27.572666 17785 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a0f54fb07dcf400b8066ec39f99fa531: Loaded TSK: 1
F20260926 19:31:27.573793 17785 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa77d77b9ad google::LogMessage::Fail() at ??:0
@ 0x7fa77d77e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa77d77b4de google::LogMessage::Flush() at ??:0
@ 0x7fa77d77ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa7814df7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa7814ce73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa7931aa1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa7931a9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa7932352a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa7932350c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa79327f5e9 std::function<>::operator()() at ??:0
@ 0x7fa7931adcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa7931abc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa793233610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa793233439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa7803c7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa7803ca91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa7803ca799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa7803a5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa78367f6db start_thread at ??:0
@ 0x7fa77c61561f clone at ??:0
W20260926 19:31:27.888357 17778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:27.888648 17778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:27.888796 17778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:27.896232 17778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:27.896335 17778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:27.896567 17778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:27.906267 17778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:46457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=43967
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:27.907601 17778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:27.908942 17778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:27.922125 17823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:27.922289 17824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:27.922704 17826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:27.923259 17778 server_base.cc:1034] running on GCE node
I20260926 19:31:27.923931 17778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:27.924921 17778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:27.926142 17778 hybrid_clock.cc:648] HybridClock initialized: now 1790451087926085 us; error 49 us; skew 500 ppm
I20260926 19:31:27.926455 17778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:27 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:27.929538 17778 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:31:27.931032 17778 webserver.cc:460] Webserver started at http://127.1.39.1:43967/ using document root <none> and password file <none>
I20260926 19:31:27.931717 17778 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:27.931866 17778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:27.937057 17778 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:27.939869 17833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:27.941174 17778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:31:27.941391 17778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "5707388c209843ff9c56f27c103c75b0"
format_stamp: "Formatted at 2026-09-26 19:30:41 on dist-test-slave-9wb0"
server_key: "0af587ffad56a458139bd8b0abb79418"
server_key_iv: "d65699405e4383ddcbaaf6329bd32dff"
server_key_version: "encryptionkey@0"
I20260926 19:31:27.942364 17778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:27.959594 17778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:27.963790 17778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:27.964083 17778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:27.965127 17778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:27.966902 17840 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260926 19:31:27.972482 17778 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 19:31:27.972631 17778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260926 19:31:27.972774 17778 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 19:31:27.975905 17778 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 19:31:27.976018 17778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260926 19:31:27.976348 17840 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Bootstrap starting.
I20260926 19:31:28.003782 17840 log.cc:824] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:28.015921 17840 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 19:31:28.016552 17840 tablet_bootstrap.cc:492] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Bootstrap complete.
I20260926 19:31:28.018301 17840 ts_tablet_manager.cc:1397] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Time spent bootstrapping tablet: real 0.042s user 0.027s sys 0.012s
I20260926 19:31:28.021410 17778 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:46457
I20260926 19:31:28.021471 17947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:46457 every 8 connection(s)
I20260926 19:31:28.022675 17778 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:28.025576 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17778
I20260926 19:31:28.026422 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:32875
--local_ip_for_outbound_sockets=127.1.39.2
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=45045
--webserver_interface=127.1.39.2
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:31:28.027842 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.025552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:49355)
0926 19:31:28.026744 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.026968 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":452,"threads_started":1}
W20260926 19:31:28.028537 17948 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:42859 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:42859: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
I20260926 19:31:28.027282 17840 raft_consensus.cc:348] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:28.028864 17840 raft_consensus.cc:729] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5707388c209843ff9c56f27c103c75b0, State: Initialized, Role: FOLLOWER
I20260926 19:31:28.029485 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.029089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:44575)
0926 19:31:28.029250 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.029352 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 19:31:28.029558 17840 consensus_queue.cc:260] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:28.030591 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.030059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:42133)
0926 19:31:28.030319 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.030423 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260926 19:31:28.030807 17948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:31:28.031351 17840 ts_tablet_manager.cc:1428] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260926 19:31:28.031589 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.031202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:33613)
0926 19:31:28.031340 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.031411 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 19:31:28.032419 17950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.032118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:51335)
0926 19:31:28.032247 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.032318 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:31:28.354560 17952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:28.354836 17952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:28.354930 17952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:28.362221 17952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:28.362334 17952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:28.362442 17952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:28.372406 17952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:32875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=45045
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:28.373704 17952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:28.375358 17952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:28.388382 17957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:28.388777 17958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:28.389319 17960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:28.389671 17952 server_base.cc:1034] running on GCE node
I20260926 19:31:28.390400 17952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:28.391500 17952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:28.392766 17952 hybrid_clock.cc:648] HybridClock initialized: now 1790451088392713 us; error 77 us; skew 500 ppm
I20260926 19:31:28.393111 17952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:28 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:28.396232 17952 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:31:28.397682 17952 webserver.cc:460] Webserver started at http://127.1.39.2:45045/ using document root <none> and password file <none>
I20260926 19:31:28.398298 17952 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:28.398396 17952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:28.403825 17952 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260926 19:31:28.406370 17967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.407639 17952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 19:31:28.407804 17952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "83aad2e2796b4914a8b817e0729e172f"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "ff5bde401748fb2ca8588b15d222da77"
server_key_iv: "e26aa6605e9881d87f984c0a91ef2d99"
server_key_version: "encryptionkey@0"
I20260926 19:31:28.408749 17952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:28.431623 17952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:28.435698 17952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:28.435979 17952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:28.436971 17952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:28.438374 17952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:28.438496 17952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.438638 17952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:28.438730 17952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.473277 17952 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:32875
I20260926 19:31:28.473397 18079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:32875 every 8 connection(s)
I20260926 19:31:28.474786 17952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:28.479637 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.477752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:51621)
0926 19:31:28.478519 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.478746 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":284,"threads_started":1}
W20260926 19:31:28.480310 18080 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:42859 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:42859: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
I20260926 19:31:28.481106 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.480788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:34429)
0926 19:31:28.480935 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.481000 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 19:31:28.481974 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.481659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:53279)
0926 19:31:28.481794 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.481874 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 19:31:28.482197 18080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:31:28.482712 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 17952
I20260926 19:31:28.483270 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:37073
--local_ip_for_outbound_sockets=127.1.39.3
--tserver_master_addrs=127.1.39.62:42859
--webserver_port=34797
--webserver_interface=127.1.39.3
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:44621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:28.809907 18083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:28.810245 18083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:28.810422 18083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:28.820894 18083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:28.821046 18083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:28.821275 18083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:28.836508 18083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:37073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=34797
--tserver_master_addrs=127.1.39.62:42859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:28.838083 18083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:28.839828 18083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:28.856585 18089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:28.856631 18091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:28.857273 18088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:28.859233 18083 server_base.cc:1034] running on GCE node
I20260926 19:31:28.859987 18083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:28.860894 18083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:28.862112 18083 hybrid_clock.cc:648] HybridClock initialized: now 1790451088862057 us; error 49 us; skew 500 ppm
I20260926 19:31:28.862427 18083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:28 dist-test-slave-9wb0 krb5kdc[16624](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451088, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:28.865624 18083 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:31:28.867192 18083 webserver.cc:460] Webserver started at http://127.1.39.3:34797/ using document root <none> and password file <none>
I20260926 19:31:28.867875 18083 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:28.868022 18083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:28.873238 18083 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:31:28.875967 18098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.877501 18083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:28.877681 18083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "2ef6ab52a6ce4c7696b96131de776864"
format_stamp: "Formatted at 2026-09-26 19:30:42 on dist-test-slave-9wb0"
server_key: "5d3583296052f59a6cbf746bdc32b68a"
server_key_iv: "74568d56b0250eab70795a8d6135c6b4"
server_key_version: "encryptionkey@0"
I20260926 19:31:28.878656 18083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:28.897775 18083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:28.901680 18083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:28.901942 18083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:28.902863 18083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:28.904136 18083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:28.904261 18083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.904390 18083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:28.904480 18083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:28.938894 18083 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:37073
I20260926 19:31:28.938979 18210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:37073 every 8 connection(s)
I20260926 19:31:28.940291 18083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:28.945380 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.943300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:47081)
0926 19:31:28.944110 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.944325 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":313,"threads_started":1}
W20260926 19:31:28.946201 18211 heartbeater.cc:644] Failed to heartbeat to 127.1.39.62:42859 (0 consecutive failures): Network error: Failed to ping master at 127.1.39.62:42859: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
I20260926 19:31:28.947165 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.946743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:35649)
0926 19:31:28.946895 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.947017 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 19:31:28.948089 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:28.947702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:56219)
0926 19:31:28.947863 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:28.947969 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 19:31:28.948338 18211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 19:31:28.948892 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18083
I20260926 19:31:29.034008 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.033237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:45307)
0926 19:31:29.033735 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:29.033817 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
I20260926 19:31:29.234731 18216 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:29.235087 18216 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:29.237038 18216 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
Sep 26 19:31:29 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
Sep 26 19:31:29 dist-test-slave-9wb0 krb5kdc[16624](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451087, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:31:29.264869 18219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.237746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:43641 (local address 127.1.39.2:32875)
0926 19:31:29.239259 (+ 1513us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:29.239269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:29.239302 (+ 33us) server_negotiation.cc:407] Connection header received
0926 19:31:29.241595 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:29.241609 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:29.242114 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:29.242483 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:29.245344 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.247276 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.250267 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.251393 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.251598 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:29.256394 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:29.256405 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:29.256423 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:29.256475 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:29.259814 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:29.261227 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:29.261236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:29.261244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:29.261492 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:29.262110 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:29.262115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:29.262119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:29.262810 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:29.262999 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:29.264039 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0926 19:31:29.264442 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1243,"threads_started":1}
I20260926 19:31:29.264868 18218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.237720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:34023 (local address 127.1.39.3:37073)
0926 19:31:29.238476 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:29.238499 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:29.239118 (+ 619us) server_negotiation.cc:407] Connection header received
0926 19:31:29.241594 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:29.241612 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:29.242114 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:29.242483 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:29.245347 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.247276 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.250267 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.251393 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.251603 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:29.256670 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:29.256676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:29.256688 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:29.256728 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:29.259821 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:29.261227 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:29.261236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:29.261244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:29.261504 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:29.262110 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:29.262115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:29.262119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:29.262810 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:29.262992 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:29.264022 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 19:31:29.264442 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":391,"threads_started":1}
I20260926 19:31:29.264858 18215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.237264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:32875 (local address 127.1.39.1:43641)
0926 19:31:29.238464 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:29.238507 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:29.239114 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:29.242681 (+ 3567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:29.242693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:29.243826 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:29.245073 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:29.245101 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.247530 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.247534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:29.250075 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:29.250091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.251684 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.251690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:29.252105 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:29.253069 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:29.253123 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:29.256084 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:29.259960 (+ 3876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:29.259971 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:29.259987 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:29.261069 (+ 1082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:29.261663 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:29.261669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:29.261673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:29.261975 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:29.262959 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:29.262967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:29.263451 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 19:31:29.263759 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 19:31:29.264332 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":519,"spinlock_wait_cycles":48000}
I20260926 19:31:29.264868 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.237368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:37073 (local address 127.1.39.1:34023)
0926 19:31:29.238998 (+ 1630us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:29.239016 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:29.239224 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:29.242681 (+ 3457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:29.242693 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:29.243826 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:29.245073 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:29.245101 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.247528 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.247534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:29.250075 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:29.250091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:29.251678 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:29.251686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:29.252102 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:29.253069 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:29.253116 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:29.256416 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:29.259981 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:29.259986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:29.259990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:29.261104 (+ 1114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:29.261663 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:29.261669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:29.261673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:29.261975 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:29.262959 (+ 984us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:29.262967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:29.263451 (+ 484us) client_negotiation.cc:769] Sending connection context
0926 19:31:29.263759 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 19:31:29.264438 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1498,"mutex_wait_us":903,"spinlock_wait_cycles":60032,"thread_start_us":1297,"threads_started":1}
W20260926 19:31:29.268548 17835 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
W20260926 19:31:29.268839 17837 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:29.268959 17837 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0; no voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f
I20260926 19:31:29.269416 18216 raft_consensus.cc:2726] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:31:29.483690 18220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.482737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:44379)
0926 19:31:29.483273 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:29.483394 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":220,"threads_started":1}
I20260926 19:31:29.949640 18221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:29.948848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:51031)
0926 19:31:29.949250 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:29.949442 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":198,"threads_started":1}
I20260926 19:31:30.035637 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:30.034947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:51977)
0926 19:31:30.035398 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:30.035478 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":240,"threads_started":1}
I20260926 19:31:30.485795 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:30.484742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:42239)
0926 19:31:30.485394 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:30.485521 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":318,"threads_started":1}
I20260926 19:31:30.951179 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:30.950436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:60615)
0926 19:31:30.950921 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:30.951006 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
I20260926 19:31:31.037408 18225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:31.036538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:41937)
0926 19:31:31.037059 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:31.037138 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":238,"threads_started":1}
I20260926 19:31:31.070060 18226 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:31.070250 18226 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:31.070966 18226 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
W20260926 19:31:31.071949 17837 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
W20260926 19:31:31.072146 17835 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:31.072314 17835 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0; no voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f
I20260926 19:31:31.072584 18226 raft_consensus.cc:2726] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:31:31.487345 18227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:31.486668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:42601)
0926 19:31:31.487114 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:31.487187 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260926 19:31:31.952661 18228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:31.951969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:35919)
0926 19:31:31.952395 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:31.952499 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":229,"threads_started":1}
I20260926 19:31:32.039067 18229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:32.038361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:53241)
0926 19:31:32.038813 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:32.038900 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":240,"threads_started":1}
I20260926 19:31:32.489861 18230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:32.488341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:54631)
0926 19:31:32.489560 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:32.489661 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":795,"threads_started":1}
I20260926 19:31:32.954416 18231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:32.953661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:54215)
0926 19:31:32.954131 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:32.954219 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":226,"threads_started":1}
I20260926 19:31:33.040938 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:33.040207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:38859)
0926 19:31:33.040687 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:33.040763 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
I20260926 19:31:33.041693 18233 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:33.041906 18233 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:33.042579 18233 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
W20260926 19:31:33.043466 17837 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
W20260926 19:31:33.043677 17835 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:33.043802 17835 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0; no voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f
I20260926 19:31:33.044070 18233 raft_consensus.cc:2726] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:31:33.491475 18234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:33.490776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:49347)
0926 19:31:33.491230 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:33.491308 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
I20260926 19:31:33.956172 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:33.955396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:41131)
0926 19:31:33.955929 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:33.956016 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":320,"threads_started":1}
I20260926 19:31:34.042620 18236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:34.041888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:45501)
0926 19:31:34.042341 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:34.042458 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20260926 19:31:34.493549 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:34.492534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:42875)
0926 19:31:34.493180 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:34.493260 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":371,"threads_started":1}
I20260926 19:31:34.957849 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:34.956990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:57543)
0926 19:31:34.957596 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:34.957671 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":328,"threads_started":1}
I20260926 19:31:35.044519 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:35.043718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:35111)
0926 19:31:35.044184 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:35.044269 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":265,"threads_started":1}
I20260926 19:31:35.495554 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:35.494716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:58637)
0926 19:31:35.495248 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:35.495338 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":226,"threads_started":1}
I20260926 19:31:35.959533 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:35.958742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:32965)
0926 19:31:35.959249 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:35.959333 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":249,"threads_started":1}
I20260926 19:31:36.046350 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:36.045581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:47681)
0926 19:31:36.046105 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:36.046183 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":245,"threads_started":1}
I20260926 19:31:36.497164 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:36.496450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:35283)
0926 19:31:36.496891 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:36.496967 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":224,"threads_started":1}
I20260926 19:31:36.525688 18244 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:36.525892 18244 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:36.526569 18244 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
W20260926 19:31:36.527468 17835 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
W20260926 19:31:36.527678 17837 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:36.527806 17837 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0; no voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f
I20260926 19:31:36.528102 18244 raft_consensus.cc:2726] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:31:36.961198 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:36.960405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:51209)
0926 19:31:36.960938 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:36.961019 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":312,"threads_started":1}
I20260926 19:31:37.047899 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:37.047227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:55903)
0926 19:31:37.047645 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:37.047721 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260926 19:31:37.498708 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:37.498017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:41265)
0926 19:31:37.498471 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:37.498550 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260926 19:31:37.962896 18248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:37.962151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:53981)
0926 19:31:37.962632 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:37.962740 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260926 19:31:38.049872 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:38.049055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:50475)
0926 19:31:38.049617 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:38.049705 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":332,"threads_started":1}
I20260926 19:31:38.500382 18250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:38.499601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:41121)
0926 19:31:38.500120 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:38.500199 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":292,"threads_started":1}
I20260926 19:31:38.964622 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:38.963830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:59615)
0926 19:31:38.964349 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:38.964431 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":240,"threads_started":1}
I20260926 19:31:39.051563 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:39.050830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:40375)
0926 19:31:39.051272 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:39.051374 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260926 19:31:39.502097 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:39.501369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:37997)
0926 19:31:39.501828 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:39.501919 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":230,"threads_started":1}
I20260926 19:31:39.966518 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:39.965683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:46701)
0926 19:31:39.966259 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:39.966342 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":318,"threads_started":1}
I20260926 19:31:40.053409 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:40.052583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:59671)
0926 19:31:40.053049 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:40.053157 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
I20260926 19:31:40.503813 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:40.503021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:55087)
0926 19:31:40.503536 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:40.503624 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260926 19:31:40.968148 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:40.967439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:46979)
0926 19:31:40.967883 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:40.967967 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260926 19:31:41.055310 18258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:41.054478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:41461)
0926 19:31:41.054997 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:41.055107 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":238,"threads_started":1}
I20260926 19:31:41.505712 18259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:41.504868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:47071)
0926 19:31:41.505425 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:41.505520 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":250,"threads_started":1}
I20260926 19:31:41.970129 18260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:41.969145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:39457)
0926 19:31:41.969864 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:41.969952 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":496,"threads_started":1}
I20260926 19:31:42.057260 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:42.056451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:45115)
0926 19:31:42.056956 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:42.057069 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260926 19:31:42.507725 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:42.506860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:55601)
0926 19:31:42.507409 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:42.507521 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":284,"threads_started":1}
I20260926 19:31:42.971961 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:42.971171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:34101)
0926 19:31:42.971682 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:42.971769 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":259,"threads_started":1}
I20260926 19:31:43.059355 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:43.058605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:57121)
0926 19:31:43.059092 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:43.059175 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":223,"threads_started":1}
I20260926 19:31:43.509791 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:43.508956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.2:39843)
0926 19:31:43.509511 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:43.509614 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260926 19:31:43.809710 18266 raft_consensus.cc:482] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:43.809918 18266 raft_consensus.cc:504] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5707388c209843ff9c56f27c103c75b0" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46457 } } peers { permanent_uuid: "83aad2e2796b4914a8b817e0729e172f" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 32875 } } peers { permanent_uuid: "2ef6ab52a6ce4c7696b96131de776864" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 37073 } }
I20260926 19:31:43.810597 18266 leader_election.cc:288] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875), 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073)
W20260926 19:31:43.811604 17835 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 83aad2e2796b4914a8b817e0729e172f (127.1.39.2:32875): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
W20260926 19:31:43.811786 17837 leader_election.cc:341] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ef6ab52a6ce4c7696b96131de776864 (127.1.39.3:37073): Not found: Tablet not found: 1ebe5c3d008e4117a309e690e8cae686
I20260926 19:31:43.811923 17837 leader_election.cc:302] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5707388c209843ff9c56f27c103c75b0; no voters: 2ef6ab52a6ce4c7696b96131de776864, 83aad2e2796b4914a8b817e0729e172f
I20260926 19:31:43.812175 18266 raft_consensus.cc:2726] T 1ebe5c3d008e4117a309e690e8cae686 P 5707388c209843ff9c56f27c103c75b0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 19:31:43.961882 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17778
I20260926 19:31:43.974067 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:43.973215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.3:51665)
0926 19:31:43.973798 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:43.973887 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":342,"threads_started":1}
I20260926 19:31:44.061349 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:44.060403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42859 (local address 127.1.39.1:39765)
0926 19:31:44.060950 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:44.061050 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.62:42859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":262,"threads_started":1}
I20260926 19:31:44.094349 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:43.963451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.1.39.1:46457)
0926 19:31:43.964101 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:43.964109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:44.074145 (+110036us) server_negotiation.cc:407] Connection header received
0926 19:31:44.074381 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:44.074390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:44.074643 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:44.074839 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:44.076359 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.077487 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.078729 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.079339 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.079483 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:44.088406 (+ 8923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:44.088423 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:44.088440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:44.088496 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:44.091393 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.092139 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.092147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.092150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.092286 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.092681 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.092687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.092691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.093057 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:44.093409 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:44.093721 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 19:31:44.094004 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":323,"threads_started":1}
I20260926 19:31:44.095618 17943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:44.186519 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17778
I20260926 19:31:44.206995 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 17952
I20260926 19:31:44.220630 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:44.207795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.1.39.2:32875)
0926 19:31:44.208312 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:44.208317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:44.208332 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:31:44.208388 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:44.208392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:44.208451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:44.208572 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:44.210074 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.210883 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.212207 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.212679 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.212813 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:44.215419 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:44.215424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:44.215427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:44.215459 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:44.218054 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.218693 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.218700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.218703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.218823 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.219192 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.219197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.219201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.219477 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:44.219751 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:44.220037 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 19:31:44.220279 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":265,"threads_started":1}
I20260926 19:31:44.221589 18075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:44.306945 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 17952
I20260926 19:31:44.325119 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18083
I20260926 19:31:44.338587 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:44.325942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59356 (local address 127.1.39.3:37073)
0926 19:31:44.326602 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:44.326606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:44.326623 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:31:44.326678 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:44.326681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:44.326751 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:44.326840 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:44.328167 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.329039 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.330499 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:44.330916 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:44.331019 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:44.333564 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:44.333569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:44.333572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:44.333604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:44.336067 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.336637 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.336642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.336644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.336785 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:44.337136 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:44.337140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:44.337143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:44.337596 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:44.337842 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:44.338075 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 19:31:44.338247 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":338,"threads_started":1}
I20260926 19:31:44.339512 18206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:44.423981 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18083
2026-09-26T19:31:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64900 ms)
[----------] 30 tests from SecurityITest (302457 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:31:44 dist-test-slave-9wb0 krb5kdc[18277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:31:44 dist-test-slave-9wb0 krb5kdc[18277](info): set up 2 sockets
Sep 26 19:31:44 dist-test-slave-9wb0 krb5kdc[18277](info): commencing operation
krb5kdc: starting...
W20260926 19:31:45.549365 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:31:45 dist-test-slave-9wb0 krb5kdc[18277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:31:45.809906 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40133
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:40133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:46.141598 18290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:46.141939 18290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:46.142113 18290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:46.149339 18290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:46.149492 18290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:46.149591 18290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:46.149694 18290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:46.149770 18290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:46.160481 18290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:40133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:46.161928 18290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:46.163261 18290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:46.175429 18295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:46.175386 18296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:46.175469 18298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:46.175995 18290 server_base.cc:1034] running on GCE node
I20260926 19:31:46.176815 18290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:46.178400 18290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:46.179661 18290 hybrid_clock.cc:648] HybridClock initialized: now 1790451106179615 us; error 52 us; skew 500 ppm
I20260926 19:31:46.180047 18290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:46 dist-test-slave-9wb0 krb5kdc[18277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:46.183169 18290 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:31:46.184777 18290 webserver.cc:460] Webserver started at http://127.1.39.62:37013/ using document root <none> and password file <none>
I20260926 19:31:46.185539 18290 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:46.185681 18290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:46.185989 18290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:46.188900 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "65444c59d42bc36355482b09e814725d"
server_key_iv: "91b0444d875ffea7fb676df250ec847c"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.189703 18290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "65444c59d42bc36355482b09e814725d"
server_key_iv: "91b0444d875ffea7fb676df250ec847c"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.194881 18290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:31:46.198246 18305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:46.199512 18290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:31:46.199685 18290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "65444c59d42bc36355482b09e814725d"
server_key_iv: "91b0444d875ffea7fb676df250ec847c"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.199877 18290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:46.231904 18290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:46.236159 18290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:46.236447 18290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:46.256310 18290 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:40133
I20260926 19:31:46.256342 18356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:40133 every 8 connection(s)
I20260926 19:31:46.257721 18290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:46.261435 18357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:46.265753 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18290
I20260926 19:31:46.265990 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:31:46.266657 1180 external_mini_cluster.cc:1428] Setting key 4f6e6673fe01e9497f620123c23e5877
I20260926 19:31:46.272302 18357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: Bootstrap starting.
I20260926 19:31:46.276357 18357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:46.277627 18357 log.cc:824] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:46.281111 18357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: No bootstrap required, opened a new log
Sep 26 19:31:46 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:46.289017 18357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } }
I20260926 19:31:46.289419 18357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:46.289547 18357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa928bd27f346f2b7e1fd014a1ec6f6, State: Initialized, Role: FOLLOWER
I20260926 19:31:46.290319 18357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } }
I20260926 19:31:46.290604 18357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:46.290757 18357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:46.290936 18357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:46.291126 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:46.268545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.1.39.62:40133)
0926 19:31:46.269446 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:46.269460 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:46.269503 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:31:46.271261 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:46.271282 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:46.271673 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:46.271976 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:46.273916 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.275715 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.277745 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.278783 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.278972 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:46.283659 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:46.283667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:46.283684 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:46.283730 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:46.286873 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:46.287555 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:46.287561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:46.287571 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:46.287747 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:46.288155 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:46.288159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:46.288161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:46.288651 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:46.288942 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:46.289411 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 19:31:46.289849 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":406,"threads_started":1}
I20260926 19:31:46.293344 18357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } }
I20260926 19:31:46.293870 18357 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fa928bd27f346f2b7e1fd014a1ec6f6; no voters:
I20260926 19:31:46.294471 18357 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:31:46.294631 18362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:46.295429 18362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [term 1 LEADER]: Becoming Leader. State: Replica: 4fa928bd27f346f2b7e1fd014a1ec6f6, State: Running, Role: LEADER
I20260926 19:31:46.296001 18362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } }
I20260926 19:31:46.296653 18357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:46.298915 18364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fa928bd27f346f2b7e1fd014a1ec6f6. Latest consensus state: current_term: 1 leader_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } } }
I20260926 19:31:46.299467 18364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:46.298942 18363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fa928bd27f346f2b7e1fd014a1ec6f6" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40133 } } }
I20260926 19:31:46.300237 18363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:46.306802 18367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:46.312032 18367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:46.322121 18367 catalog_manager.cc:1347] Generated new cluster ID: 05c0dd942c9441d5b96f7408cd627187
I20260926 19:31:46.322247 18367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:46.355412 18367 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:31:46.357285 18367 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:46.367066 18367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: Generated new TSK 0
I20260926 19:31:46.368008 18367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:31:46.433037 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:46.769836 18383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:46.770102 18383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:46.770211 18383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:46.777444 18383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:46.777540 18383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:46.777654 18383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:46.786985 18383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:46.788144 18383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:46.789399 18383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:46.802852 18388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:46.802924 18391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:46.802953 18389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:46.803934 18383 server_base.cc:1034] running on GCE node
I20260926 19:31:46.804629 18383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:46.805653 18383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:46.806866 18383 hybrid_clock.cc:648] HybridClock initialized: now 1790451106806796 us; error 53 us; skew 500 ppm
I20260926 19:31:46.807183 18383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:46 dist-test-slave-9wb0 krb5kdc[18277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:46.810287 18383 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:31:46.811743 18383 webserver.cc:460] Webserver started at http://127.1.39.1:39463/ using document root <none> and password file <none>
I20260926 19:31:46.812503 18383 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:46.812654 18383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:46.813056 18383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:46.816118 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "8e11c29828174ff7b1306f1bd149c169"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "d6c19eec420c8fc42e7cfeaef4641c7c"
server_key_iv: "0bee56a0c352bb212bf6f433749234f4"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.816973 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "8e11c29828174ff7b1306f1bd149c169"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "d6c19eec420c8fc42e7cfeaef4641c7c"
server_key_iv: "0bee56a0c352bb212bf6f433749234f4"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.822058 18383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 19:31:46.825557 18398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:46.826876 18383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:31:46.827062 18383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "8e11c29828174ff7b1306f1bd149c169"
format_stamp: "Formatted at 2026-09-26 19:31:46 on dist-test-slave-9wb0"
server_key: "d6c19eec420c8fc42e7cfeaef4641c7c"
server_key_iv: "0bee56a0c352bb212bf6f433749234f4"
server_key_version: "encryptionkey@0"
I20260926 19:31:46.827316 18383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:46.845669 18383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:46.849853 18383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:46.850142 18383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:46.851114 18383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:46.852478 18383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:46.852550 18383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:46.852716 18383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:46.852777 18383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:46.887187 18383 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:42133
I20260926 19:31:46.887239 18510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:42133 every 8 connection(s)
I20260926 19:31:46.888559 18383 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:46.898686 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18383
I20260926 19:31:46.898904 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:31:46.899708 1180 external_mini_cluster.cc:1428] Setting key fcebb4c66826a5ee0456d484de4e3656
Sep 26 19:31:46 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:46.908684 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:46.891169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:43283 (local address 127.1.39.62:40133)
0926 19:31:46.891613 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:46.891617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:46.892530 (+ 913us) server_negotiation.cc:407] Connection header received
0926 19:31:46.894398 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:46.894402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:46.894472 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:46.894571 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:46.896091 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.896877 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.898580 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.899245 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.899387 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:46.903057 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:46.903062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:46.903065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:46.903095 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:46.905734 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:46.906428 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:46.906434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:46.906444 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:46.906561 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:46.906999 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:46.907008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:46.907010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:46.907256 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:46.907424 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:46.908106 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 19:31:46.908332 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":226,"threads_started":1}
I20260926 19:31:46.909725 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:46.891688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40133 (local address 127.1.39.1:43283)
0926 19:31:46.892363 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:46.892404 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:46.894120 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:46.894767 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:46.894774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:46.895284 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:46.895930 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:46.895947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.897028 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.897031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:46.898402 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:46.898414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:46.899453 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:46.899458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:46.899643 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:46.900339 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:46.900365 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:46.902833 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:46.905876 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:46.905885 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:46.905900 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:46.906301 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:46.906681 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:46.906686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:46.906690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:46.906870 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:46.907384 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:46.907393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:46.907730 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 19:31:46.907990 (+ 260us) client_negotiation.cc:240] Negotiation successful
0926 19:31:46.908504 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":284,"threads_started":1}
I20260926 19:31:46.911264 18511 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40133
I20260926 19:31:46.911620 18511 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:46.912396 18511 heartbeater.cc:511] Master 127.1.39.62:40133 requested a full tablet report, sending...
I20260926 19:31:46.914858 18322 ts_manager.cc:194] Registered new tserver with Master: 8e11c29828174ff7b1306f1bd149c169 (127.1.39.1:42133)
I20260926 19:31:46.917016 18322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:43283
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:31:46.960605 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:47.292681 18517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:47.292956 18517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:47.293049 18517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:47.300498 18517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:47.300611 18517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:47.300725 18517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:47.310740 18517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:47.312043 18517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:47.313504 18517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:47.327185 18525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:47.327185 18523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:47.327621 18522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:47.330916 18517 server_base.cc:1034] running on GCE node
I20260926 19:31:47.331581 18517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:47.332647 18517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:47.333847 18517 hybrid_clock.cc:648] HybridClock initialized: now 1790451107333801 us; error 35 us; skew 500 ppm
I20260926 19:31:47.334165 18517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:47 dist-test-slave-9wb0 krb5kdc[18277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:47.337425 18517 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:31:47.338855 18517 webserver.cc:460] Webserver started at http://127.1.39.2:38765/ using document root <none> and password file <none>
I20260926 19:31:47.339573 18517 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:47.339715 18517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:47.340034 18517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:47.343071 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "c2731a0690fb408a8d3816a636d85f02"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "3db88690c59af147de41a135e6088250"
server_key_iv: "e9752bb760595d04655f35202cdfbb95"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.343874 18517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "c2731a0690fb408a8d3816a636d85f02"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "3db88690c59af147de41a135e6088250"
server_key_iv: "e9752bb760595d04655f35202cdfbb95"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.349049 18517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 19:31:47.352538 18532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.353894 18517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:47.354084 18517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "c2731a0690fb408a8d3816a636d85f02"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "3db88690c59af147de41a135e6088250"
server_key_iv: "e9752bb760595d04655f35202cdfbb95"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.354300 18517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:47.375573 18517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:47.379987 18517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:47.380286 18517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:47.381237 18517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:47.382532 18517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:47.382644 18517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.382792 18517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:47.382865 18517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.417898 18517 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:35661
I20260926 19:31:47.417954 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:35661 every 8 connection(s)
I20260926 19:31:47.419237 18517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:47.427172 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18517
I20260926 19:31:47.427392 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:31:47.427868 1180 external_mini_cluster.cc:1428] Setting key 1792acbaefb0db6df46b8b1fcc22a87a
Sep 26 19:31:47 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:47.440337 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:47.421790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:38951 (local address 127.1.39.62:40133)
0926 19:31:47.422210 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:47.422215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:47.423205 (+ 990us) server_negotiation.cc:407] Connection header received
0926 19:31:47.425047 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:47.425054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:47.425140 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:47.425266 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:47.426712 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.427536 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.429280 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.429782 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.429936 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:47.433476 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:47.433484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:47.433492 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:47.433539 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:47.437172 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.437953 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.437961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.437968 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.438133 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.438554 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.438562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.438567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.438944 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:47.439112 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:47.439767 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 19:31:47.440037 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":216,"threads_started":1}
I20260926 19:31:47.441552 18648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:47.422364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40133 (local address 127.1.39.2:38951)
0926 19:31:47.423042 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:47.423078 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:47.424789 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:47.425513 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:47.425521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:47.426015 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:47.426545 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:47.426562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.427703 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.427707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:47.429127 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:47.429140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.430024 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.430032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:47.430275 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:47.430987 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:47.431027 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:47.433213 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:47.437384 (+ 4171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:47.437391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:47.437405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:47.437826 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:47.438256 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:47.438259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:47.438261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:47.438437 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:47.439061 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:47.439067 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:47.439387 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 19:31:47.439608 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 19:31:47.440028 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":271,"threads_started":1}
I20260926 19:31:47.443222 18645 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40133
I20260926 19:31:47.443558 18645 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:47.444299 18645 heartbeater.cc:511] Master 127.1.39.62:40133 requested a full tablet report, sending...
I20260926 19:31:47.445858 18322 ts_manager.cc:194] Registered new tserver with Master: c2731a0690fb408a8d3816a636d85f02 (127.1.39.2:35661)
I20260926 19:31:47.446687 18322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:38951
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:31:47.489166 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:41161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:47.818243 18651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:47.818539 18651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:47.818723 18651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:47.825971 18651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:47.826094 18651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:47.826311 18651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:47.836242 18651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:41161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:47.837639 18651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:47.838927 18651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:47.852154 18656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:47.852177 18657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:47.852355 18659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:47.853013 18651 server_base.cc:1034] running on GCE node
I20260926 19:31:47.853674 18651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:47.854621 18651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:47.855850 18651 hybrid_clock.cc:648] HybridClock initialized: now 1790451107855817 us; error 58 us; skew 500 ppm
I20260926 19:31:47.856173 18651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:47 dist-test-slave-9wb0 krb5kdc[18277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:47.859530 18651 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:31:47.861044 18651 webserver.cc:460] Webserver started at http://127.1.39.3:38107/ using document root <none> and password file <none>
I20260926 19:31:47.861843 18651 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:47.861948 18651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:47.862283 18651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:47.865187 18651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "a6980777be4748ed819c510b53283d90"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "61ee27f01ed90b6b5127c3d508621e0d"
server_key_iv: "07c73f0fdcb7fe156e91d873633c3850"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.866050 18651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "a6980777be4748ed819c510b53283d90"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "61ee27f01ed90b6b5127c3d508621e0d"
server_key_iv: "07c73f0fdcb7fe156e91d873633c3850"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.871174 18651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:31:47.874610 18666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.875960 18651 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:31:47.876148 18651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "a6980777be4748ed819c510b53283d90"
format_stamp: "Formatted at 2026-09-26 19:31:47 on dist-test-slave-9wb0"
server_key: "61ee27f01ed90b6b5127c3d508621e0d"
server_key_iv: "07c73f0fdcb7fe156e91d873633c3850"
server_key_version: "encryptionkey@0"
I20260926 19:31:47.876353 18651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:47.891010 18651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:47.895434 18651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:47.895732 18651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:47.896699 18651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:47.898279 18651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:47.898355 18651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.898530 18651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:47.898584 18651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:47.920105 18511 heartbeater.cc:503] Master 127.1.39.62:40133 was elected leader, sending a full tablet report...
I20260926 19:31:47.931690 18651 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:34847
I20260926 19:31:47.931783 18778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:34847 every 8 connection(s)
I20260926 19:31:47.932969 18651 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:47.936354 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18651
I20260926 19:31:47.936672 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:31:47.937315 1180 external_mini_cluster.cc:1428] Setting key 4bc40dda34f321417b0de9ff22483427
Sep 26 19:31:47 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:47.953799 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:47.935538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:42431 (local address 127.1.39.62:40133)
0926 19:31:47.935725 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:47.935730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:47.937180 (+ 1450us) server_negotiation.cc:407] Connection header received
0926 19:31:47.939519 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:47.939525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:47.939621 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:47.939700 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:47.941507 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.942275 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.943801 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.944198 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.944293 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:47.947662 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:47.947667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:47.947670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:47.947704 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:47.950379 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.951231 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.951236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.951239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.951392 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.951903 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.951908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.951911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.952254 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:47.952426 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:47.953395 (+ 969us) server_negotiation.cc:299] Negotiation successful
0926 19:31:47.953574 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:31:47.954783 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:47.936036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40133 (local address 127.1.39.3:42431)
0926 19:31:47.936984 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:47.937036 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:47.939250 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:47.939977 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:47.939990 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:47.940715 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:47.941346 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:47.941371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.942420 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.942424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:47.943668 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:47.943681 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.944382 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.944387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:47.944597 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:47.945269 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:47.945337 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:47.947459 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:47.950521 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:47.950527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:47.950540 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:47.951086 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:47.951512 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:47.951517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:47.951520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:47.951776 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:47.952460 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:47.952469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:47.952887 (+ 418us) client_negotiation.cc:769] Sending connection context
0926 19:31:47.953235 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 19:31:47.953708 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":426,"threads_started":1}
I20260926 19:31:47.956067 18779 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40133
I20260926 19:31:47.956341 18779 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:47.957006 18779 heartbeater.cc:511] Master 127.1.39.62:40133 requested a full tablet report, sending...
I20260926 19:31:47.958477 18322 ts_manager.cc:194] Registered new tserver with Master: a6980777be4748ed819c510b53283d90 (127.1.39.3:34847)
I20260926 19:31:47.959290 18322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:42431
I20260926 19:31:47.965811 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:31:47.982758 18647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:47.970334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.1.39.62:40133)
0926 19:31:47.970504 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:47.970508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:47.970903 (+ 395us) server_negotiation.cc:407] Connection header received
0926 19:31:47.971040 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:47.971044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:47.971110 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:47.971190 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:47.972664 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.973518 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.974826 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:47.975209 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:47.975305 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:47.977656 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:47.977661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:47.977665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:47.977698 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:47.980344 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.980968 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.980972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.980974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.981110 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:47.981536 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:47.981539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:47.981541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:47.981846 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:47.982040 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:47.982327 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 19:31:47.982483 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 19:31:47.987026 18322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:31:47.994364 18322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:31:48.036494 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.006726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.1.39.3:34847)
0926 19:31:48.028181 (+ 21455us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.028189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.028216 (+ 27us) server_negotiation.cc:407] Connection header received
0926 19:31:48.028292 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.028297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.028462 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.028655 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.030293 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.031922 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.034068 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.035304 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.035442 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.035666 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.035959 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.036165 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21281,"thread_start_us":21144,"threads_started":1}
I20260926 19:31:48.036547 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.006428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:34847 (local address 127.0.0.1:50896)
0926 19:31:48.026345 (+ 19917us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.026383 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.026556 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.028930 (+ 2374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.028938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.028998 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.030088 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.030100 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.032070 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.032076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.033919 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.033931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.035499 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.035506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.035703 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.035754 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.035876 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.036060 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19638,"mutex_wait_us":675,"spinlock_wait_cycles":33920,"thread_start_us":19505,"threads_started":1}
I20260926 19:31:48.037361 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.006763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35661 (local address 127.0.0.1:58418)
0926 19:31:48.027795 (+ 21032us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.027814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.027940 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.028930 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.028938 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.028990 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.029516 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.029534 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.031923 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.031929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.034421 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.034436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.036394 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.036402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.036632 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.036685 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.036767 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.036880 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20908,"mutex_wait_us":22,"spinlock_wait_cycles":896,"thread_start_us":20750,"threads_started":1}
I20260926 19:31:48.037372 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.006944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.1.39.2:35661)
0926 19:31:48.027133 (+ 20189us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.027146 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.028090 (+ 944us) server_negotiation.cc:407] Connection header received
0926 19:31:48.028203 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.028212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.028460 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.028655 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.029826 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.031784 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.034588 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.035950 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.036063 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.036244 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.036874 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.037047 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19941,"thread_start_us":20195,"threads_started":1}
I20260926 19:31:48.038285 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.007441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.1.39.1:42133)
0926 19:31:48.029948 (+ 22507us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.029954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.029976 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:31:48.030047 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.030051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.030276 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.030493 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.031061 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.032572 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.034968 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.036215 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.036332 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.036539 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.037787 (+ 1248us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.037953 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22358,"thread_start_us":22186,"threads_started":1}
I20260926 19:31:48.037490 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.007271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:42133 (local address 127.0.0.1:33948)
0926 19:31:48.029536 (+ 22265us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.029555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.029669 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.030487 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.030492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.030518 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.030868 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.030881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.032718 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.032728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.034822 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.034832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.036688 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.036694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.036871 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.036904 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.036990 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.037116 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22130,"thread_start_us":21923,"threads_started":1}
I20260926 19:31:48.041126 18446 tablet_service.cc:1467] Processing CreateTablet for tablet cba366dc58c044d981af94291b3af03c (DEFAULT_TABLE table=test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:48.041211 18580 tablet_service.cc:1467] Processing CreateTablet for tablet cba366dc58c044d981af94291b3af03c (DEFAULT_TABLE table=test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:48.041137 18714 tablet_service.cc:1467] Processing CreateTablet for tablet cba366dc58c044d981af94291b3af03c (DEFAULT_TABLE table=test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:48.042426 18446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba366dc58c044d981af94291b3af03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:48.042464 18714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba366dc58c044d981af94291b3af03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:48.042464 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba366dc58c044d981af94291b3af03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:48.053992 18797 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Bootstrap starting.
I20260926 19:31:48.057089 18798 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Bootstrap starting.
I20260926 19:31:48.057940 18799 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Bootstrap starting.
I20260926 19:31:48.058856 18797 tablet_bootstrap.cc:654] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:48.059883 18797 log.cc:824] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:48.061896 18798 tablet_bootstrap.cc:654] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:48.062026 18799 tablet_bootstrap.cc:654] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:48.062793 18797 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: No bootstrap required, opened a new log
I20260926 19:31:48.063063 18797 ts_tablet_manager.cc:1397] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260926 19:31:48.063206 18798 log.cc:824] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:48.063206 18799 log.cc:824] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:48.066309 18798 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: No bootstrap required, opened a new log
I20260926 19:31:48.066363 18799 tablet_bootstrap.cc:492] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: No bootstrap required, opened a new log
I20260926 19:31:48.066598 18798 ts_tablet_manager.cc:1397] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 19:31:48.066601 18799 ts_tablet_manager.cc:1397] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260926 19:31:48.074467 18798 raft_consensus.cc:348] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.074883 18798 raft_consensus.cc:374] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:48.074990 18798 raft_consensus.cc:729] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2731a0690fb408a8d3816a636d85f02, State: Initialized, Role: FOLLOWER
I20260926 19:31:48.074972 18797 raft_consensus.cc:348] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.075389 18797 raft_consensus.cc:374] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:48.075476 18797 raft_consensus.cc:729] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e11c29828174ff7b1306f1bd149c169, State: Initialized, Role: FOLLOWER
I20260926 19:31:48.075693 18798 consensus_queue.cc:260] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.076058 18797 consensus_queue.cc:260] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.077062 18645 heartbeater.cc:503] Master 127.1.39.62:40133 was elected leader, sending a full tablet report...
I20260926 19:31:48.077600 18798 ts_tablet_manager.cc:1428] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260926 19:31:48.079281 18799 raft_consensus.cc:348] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.079825 18799 raft_consensus.cc:374] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:48.079953 18799 raft_consensus.cc:729] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6980777be4748ed819c510b53283d90, State: Initialized, Role: FOLLOWER
I20260926 19:31:48.080659 18799 consensus_queue.cc:260] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.082032 18779 heartbeater.cc:503] Master 127.1.39.62:40133 was elected leader, sending a full tablet report...
I20260926 19:31:48.082903 18799 ts_tablet_manager.cc:1428] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260926 19:31:48.084559 18797 ts_tablet_manager.cc:1428] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Time spent starting tablet: real 0.021s user 0.019s sys 0.002s
W20260926 19:31:48.141026 18512 tablet.cc:2369] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:48.171265 18646 tablet.cc:2369] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:48.184628 18780 tablet.cc:2369] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:31:48.418658 18803 raft_consensus.cc:482] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:48.419001 18803 raft_consensus.cc:504] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.420819 18803 leader_election.cc:288] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6980777be4748ed819c510b53283d90 (127.1.39.3:34847), 8e11c29828174ff7b1306f1bd149c169 (127.1.39.1:42133)
I20260926 19:31:48.429602 18805 raft_consensus.cc:482] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:48.429834 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.421552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:55515 (local address 127.1.39.3:34847)
0926 19:31:48.421798 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.421806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.423115 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 19:31:48.423343 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.423348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.423408 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.423511 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.424491 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.425584 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.427947 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.428728 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.428882 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.428991 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.429403 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.429548 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:31:48.429906 18805 raft_consensus.cc:504] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.430398 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.422159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:56873 (local address 127.1.39.1:42133)
0926 19:31:48.422398 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.422407 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.423306 (+ 899us) server_negotiation.cc:407] Connection header received
0926 19:31:48.423557 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.423563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.423641 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.423724 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.424388 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.425154 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.427729 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.428985 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.429127 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.429239 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.429612 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.429851 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:31:48.430783 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.421003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:34847 (local address 127.1.39.2:55515)
0926 19:31:48.422984 (+ 1981us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.423010 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.423184 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.423853 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.423856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.423888 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.424344 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.424357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.425716 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.425723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.427803 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.427812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.428944 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.428950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.429167 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.429215 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.430120 (+ 905us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.430273 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1821,"spinlock_wait_cycles":13568,"thread_start_us":1901,"threads_started":1}
I20260926 19:31:48.431445 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.421259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:42133 (local address 127.1.39.2:56873)
0926 19:31:48.423210 (+ 1951us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.423226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.423401 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.423760 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.423765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.423788 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.424204 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.424214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.425314 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.425320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.427561 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.427575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.429221 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.429226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.429443 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.429474 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.430739 (+ 1265us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.431054 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"thread_start_us":1310,"threads_started":1}
I20260926 19:31:48.431743 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "c2731a0690fb408a8d3816a636d85f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6980777be4748ed819c510b53283d90" is_pre_election: true
I20260926 19:31:48.432253 18734 raft_consensus.cc:2454] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2731a0690fb408a8d3816a636d85f02 in term 0.
I20260926 19:31:48.432639 18466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "c2731a0690fb408a8d3816a636d85f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e11c29828174ff7b1306f1bd149c169" is_pre_election: true
I20260926 19:31:48.433264 18805 leader_election.cc:288] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2731a0690fb408a8d3816a636d85f02 (127.1.39.2:35661), 8e11c29828174ff7b1306f1bd149c169 (127.1.39.1:42133)
I20260926 19:31:48.433219 18466 raft_consensus.cc:2454] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2731a0690fb408a8d3816a636d85f02 in term 0.
I20260926 19:31:48.434063 18536 leader_election.cc:302] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e11c29828174ff7b1306f1bd149c169, c2731a0690fb408a8d3816a636d85f02; no voters:
I20260926 19:31:48.434511 18803 raft_consensus.cc:2781] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:31:48.434648 18803 raft_consensus.cc:482] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:31:48.434736 18803 raft_consensus.cc:3037] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:48.437908 18803 raft_consensus.cc:504] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.438966 18734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "c2731a0690fb408a8d3816a636d85f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6980777be4748ed819c510b53283d90"
I20260926 19:31:48.438954 18466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "c2731a0690fb408a8d3816a636d85f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e11c29828174ff7b1306f1bd149c169"
I20260926 19:31:48.439173 18734 raft_consensus.cc:3037] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:48.439172 18466 raft_consensus.cc:3037] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:48.439508 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.432215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:42133 (local address 127.1.39.3:41599)
0926 19:31:48.432718 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.432740 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.432855 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.433397 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.433402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.433426 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.433630 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.433643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.435050 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.435058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.437092 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.437113 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.438734 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.438741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.438924 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.438965 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.439030 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.439189 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":24}
I20260926 19:31:48.440457 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.432547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:41599 (local address 127.1.39.1:42133)
0926 19:31:48.432699 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.432705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.432974 (+ 269us) server_negotiation.cc:407] Connection header received
0926 19:31:48.433036 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.433041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.433112 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.433209 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.433799 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.434870 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.437258 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.438552 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.439836 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.439968 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.440021 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.440168 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:31:48.441046 18465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "a6980777be4748ed819c510b53283d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e11c29828174ff7b1306f1bd149c169" is_pre_election: true
I20260926 19:31:48.441639 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.433820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:47489 (local address 127.1.39.2:35661)
0926 19:31:48.434038 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.434050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.435008 (+ 958us) server_negotiation.cc:407] Connection header received
0926 19:31:48.435277 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.435282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.435345 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.435448 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:48.436105 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.436826 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.439252 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.440430 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.440548 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.440657 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.441254 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.441408 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:31:48.442317 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.433016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:35661 (local address 127.1.39.3:47489)
0926 19:31:48.434917 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:48.434935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:48.435111 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:48.435461 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:48.435470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:48.435494 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:48.435919 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.435932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.436975 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.436981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.439088 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:48.439099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.440842 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.440848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:48.441050 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.441090 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:31:48.441788 (+ 698us) client_negotiation.cc:240] Negotiation successful
0926 19:31:48.441931 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1750,"mutex_wait_us":2,"thread_start_us":1489,"threads_started":1}
I20260926 19:31:48.442519 18803 leader_election.cc:288] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [CANDIDATE]: Term 1 election: Requested vote from peers a6980777be4748ed819c510b53283d90 (127.1.39.3:34847), 8e11c29828174ff7b1306f1bd149c169 (127.1.39.1:42133)
I20260926 19:31:48.442571 18466 raft_consensus.cc:2454] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2731a0690fb408a8d3816a636d85f02 in term 1.
I20260926 19:31:48.442975 18536 leader_election.cc:302] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e11c29828174ff7b1306f1bd149c169, c2731a0690fb408a8d3816a636d85f02; no voters:
I20260926 19:31:48.443171 18734 raft_consensus.cc:2454] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2731a0690fb408a8d3816a636d85f02 in term 1.
I20260926 19:31:48.443257 18803 raft_consensus.cc:2781] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:48.443204 18600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba366dc58c044d981af94291b3af03c" candidate_uuid: "a6980777be4748ed819c510b53283d90" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2731a0690fb408a8d3816a636d85f02" is_pre_election: true
I20260926 19:31:48.443779 18803 raft_consensus.cc:686] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 1 LEADER]: Becoming Leader. State: Replica: c2731a0690fb408a8d3816a636d85f02, State: Running, Role: LEADER
I20260926 19:31:48.444268 18803 consensus_queue.cc:237] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } }
I20260926 19:31:48.448257 18669 leader_election.cc:302] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6980777be4748ed819c510b53283d90; no voters: 8e11c29828174ff7b1306f1bd149c169, c2731a0690fb408a8d3816a636d85f02
I20260926 19:31:48.448650 18805 raft_consensus.cc:2726] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 19:31:48.449478 18321 catalog_manager.cc:5495] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2731a0690fb408a8d3816a636d85f02 (127.1.39.2). New cstate: current_term: 1 leader_uuid: "c2731a0690fb408a8d3816a636d85f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2731a0690fb408a8d3816a636d85f02" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 35661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 } health_report { overall_health: UNKNOWN } } }
I20260926 19:31:48.473419 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.467538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.1.39.2:35661)
0926 19:31:48.467732 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.467739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.467929 (+ 190us) server_negotiation.cc:407] Connection header received
0926 19:31:48.468201 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.468205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.468286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.468438 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:48.468834 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.469639 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.471138 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.471559 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.471696 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.472217 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:48.472758 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:31:48.472914 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.473018 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.473110 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:31:48.481804 18466 raft_consensus.cc:1261] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 1 FOLLOWER]: Refusing update from remote peer c2731a0690fb408a8d3816a636d85f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:48.481859 18734 raft_consensus.cc:1261] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 1 FOLLOWER]: Refusing update from remote peer c2731a0690fb408a8d3816a636d85f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:48.482645 18809 consensus_queue.cc:1035] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e11c29828174ff7b1306f1bd149c169" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 42133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:48.483016 18803 consensus_queue.cc:1035] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6980777be4748ed819c510b53283d90" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:48.505492 18813 mvcc.cc:205] Tried to move back new op lower bound from 7333687740331675648 to 7333687740191522816. Current Snapshot: MvccSnapshot[applied={T|T < 7333687740331675648}]
I20260926 19:31:48.517732 18321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:48.518046 18321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:48.521552 18321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4fa928bd27f346f2b7e1fd014a1ec6f6: Sending DeleteTablet for 3 replicas of tablet cba366dc58c044d981af94291b3af03c
I20260926 19:31:48.522704 18714 tablet_service.cc:1514] Processing DeleteTablet for tablet cba366dc58c044d981af94291b3af03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:48 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:50896
I20260926 19:31:48.523000 18446 tablet_service.cc:1514] Processing DeleteTablet for tablet cba366dc58c044d981af94291b3af03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:48 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:33948
I20260926 19:31:48.523479 18580 tablet_service.cc:1514] Processing DeleteTablet for tablet cba366dc58c044d981af94291b3af03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:48 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:58418
I20260926 19:31:48.526242 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18383
I20260926 19:31:48.536295 18823 tablet_replica.cc:330] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: stopping tablet replica
I20260926 19:31:48.536901 18823 raft_consensus.cc:2229] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:48.537638 18823 raft_consensus.cc:2258] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:48.541005 18823 ts_tablet_manager.cc:1905] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:48.551874 18823 ts_tablet_manager.cc:1918] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:48.552101 18823 log.cc:1196] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/cba366dc58c044d981af94291b3af03c
I20260926 19:31:48.552851 18823 ts_tablet_manager.cc:1939] T cba366dc58c044d981af94291b3af03c P 8e11c29828174ff7b1306f1bd149c169: Deleting consensus metadata
I20260926 19:31:48.555306 18824 tablet_replica.cc:330] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: stopping tablet replica
I20260926 19:31:48.555425 18309 catalog_manager.cc:4841] TS 8e11c29828174ff7b1306f1bd149c169 (127.1.39.1:42133): tablet cba366dc58c044d981af94291b3af03c (table test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]) successfully deleted
I20260926 19:31:48.555780 18824 raft_consensus.cc:2229] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:31:48.556452 18824 raft_consensus.cc:2258] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:48.557453 18822 tablet_replica.cc:330] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: stopping tablet replica
I20260926 19:31:48.557897 18822 raft_consensus.cc:2229] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:48.558386 18822 raft_consensus.cc:2258] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:48.560683 18824 ts_tablet_manager.cc:1905] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:31:48 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:31:48.567601 18822 ts_tablet_manager.cc:1905] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:48.568423 18824 ts_tablet_manager.cc:1918] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:48.568583 18824 log.cc:1196] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/cba366dc58c044d981af94291b3af03c
I20260926 19:31:48.569118 18824 ts_tablet_manager.cc:1939] T cba366dc58c044d981af94291b3af03c P c2731a0690fb408a8d3816a636d85f02: Deleting consensus metadata
I20260926 19:31:48.570987 18308 catalog_manager.cc:4841] TS c2731a0690fb408a8d3816a636d85f02 (127.1.39.2:35661): tablet cba366dc58c044d981af94291b3af03c (table test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]) successfully deleted
I20260926 19:31:48.574265 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.527618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.1.39.1:42133)
0926 19:31:48.527802 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.527809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.552327 (+ 24518us) server_negotiation.cc:407] Connection header received
0926 19:31:48.552854 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.552861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.552953 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.553024 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:48.556879 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.558084 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.559827 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.560278 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.560429 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.565125 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:48.565137 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:48.565151 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:48.565195 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:48.569752 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.570786 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.570796 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.570800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.570974 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.571521 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.571530 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.571537 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.572042 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:48.572367 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.573603 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.573847 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:31:48.575281 18506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:48.575249 18822 ts_tablet_manager.cc:1918] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:48.575429 18822 log.cc:1196] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/cba366dc58c044d981af94291b3af03c
I20260926 19:31:48.576179 18822 ts_tablet_manager.cc:1939] T cba366dc58c044d981af94291b3af03c P a6980777be4748ed819c510b53283d90: Deleting consensus metadata
I20260926 19:31:48.578442 18307 catalog_manager.cc:4841] TS a6980777be4748ed819c510b53283d90 (127.1.39.3:34847): tablet cba366dc58c044d981af94291b3af03c (table test-table [id=8b40ddbc9e4b4dea9cfc7d0e9c49ceec]) successfully deleted
I20260926 19:31:48.666747 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18383
I20260926 19:31:48.692818 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18517
Sep 26 19:31:48 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:31:48.708822 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.693606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58432 (local address 127.1.39.2:35661)
0926 19:31:48.693809 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.693817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.693837 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:31:48.694044 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.694049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.694129 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.694242 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:48.695494 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.696247 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.697692 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.698205 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.698337 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.702793 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:48.702807 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:48.702823 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:48.702855 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:48.706070 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.706770 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.706781 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.706783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.706940 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.707324 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.707330 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.707333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.707690 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:48.708011 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.708273 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.708487 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:31:48.709546 18640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:48.822883 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18517
I20260926 19:31:48.842525 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18651
Sep 26 19:31:48 dist-test-slave-9wb0 krb5kdc[18277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:31:48.858135 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:48.843264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.1.39.3:34847)
0926 19:31:48.843500 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:48.843505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:48.843526 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:31:48.843674 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:48.843679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:48.843771 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:48.843879 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:48.845242 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.846037 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.847300 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:48.847709 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:48.847849 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:48.852333 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:48.852349 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:48.852366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:48.852402 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:48.855142 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.855841 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.855848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.855851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.855992 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:48.856403 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:48.856408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:48.856411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:48.856788 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:48.857105 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:48.857554 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 19:31:48.857775 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 19:31:48.858748 18774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:48.968398 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18651
I20260926 19:31:48.985424 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18290
I20260926 19:31:48.986553 18352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:49.101172 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18290
2026-09-26T19:31:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4680 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T19:31:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:31:49Z Disabled control of system clock
I20260926 19:31:49.180810 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46809
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:46809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:31:49.504400 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:49.504700 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:49.504782 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:49.512012 18833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:49.512122 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:49.512161 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:49.512216 18833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:49.512260 18833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:49.521415 18833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:46809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:46809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:49.522760 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:49.524510 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:49.536339 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:49.536487 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:49.536535 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:49.537600 18833 server_base.cc:1034] running on GCE node
I20260926 19:31:49.538386 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:49.539758 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:49.540987 18833 hybrid_clock.cc:648] HybridClock initialized: now 1790451109540932 us; error 67 us; skew 500 ppm
I20260926 19:31:49.541397 18833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:49.547791 18833 webserver.cc:460] Webserver started at http://127.1.39.62:41777/ using document root <none> and password file <none>
I20260926 19:31:49.548506 18833 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:49.548645 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:49.548964 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:49.551920 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "bda7f8046a9f469eb75ac50e4648a90c"
format_stamp: "Formatted at 2026-09-26 19:31:49 on dist-test-slave-9wb0"
server_key: "51244745aef5e96b90ef122f4f6dc869"
server_key_iv: "73f6cdf51c91270ee3d7ade7bc46c4da"
server_key_version: "encryptionkey@0"
I20260926 19:31:49.552717 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "bda7f8046a9f469eb75ac50e4648a90c"
format_stamp: "Formatted at 2026-09-26 19:31:49 on dist-test-slave-9wb0"
server_key: "51244745aef5e96b90ef122f4f6dc869"
server_key_iv: "73f6cdf51c91270ee3d7ade7bc46c4da"
server_key_version: "encryptionkey@0"
I20260926 19:31:49.557649 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:31:49.560904 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:49.562287 18833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:49.562472 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "bda7f8046a9f469eb75ac50e4648a90c"
format_stamp: "Formatted at 2026-09-26 19:31:49 on dist-test-slave-9wb0"
server_key: "51244745aef5e96b90ef122f4f6dc869"
server_key_iv: "73f6cdf51c91270ee3d7ade7bc46c4da"
server_key_version: "encryptionkey@0"
I20260926 19:31:49.562680 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:49.576432 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:49.577232 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:49.577546 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:49.595221 18833 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:46809
I20260926 19:31:49.595240 18898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:46809 every 8 connection(s)
I20260926 19:31:49.596594 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:49.600111 18899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:49.603082 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18833
I20260926 19:31:49.603256 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:31:49.603818 1180 external_mini_cluster.cc:1428] Setting key 7b0e6d6f84dfc341bac538056547e243
I20260926 19:31:49.612241 18899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: Bootstrap starting.
I20260926 19:31:49.615849 18899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:49.616995 18899 log.cc:824] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:49.617568 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:49.605738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.1.39.62:46809)
0926 19:31:49.606769 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:49.606786 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:49.606830 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:31:49.608479 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:49.608503 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:49.608948 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:49.609260 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:49.609846 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:49.611118 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:49.613015 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:49.613768 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:49.613903 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:49.615152 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:49.615159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:49.615172 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:49.615691 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:49.615786 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:49.615806 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:49.615893 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:49.616159 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 19:31:49.616500 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":397,"threads_started":1}
I20260926 19:31:49.619817 18899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: No bootstrap required, opened a new log
I20260926 19:31:49.627285 18899 raft_consensus.cc:348] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } }
I20260926 19:31:49.627657 18899 raft_consensus.cc:374] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:49.627763 18899 raft_consensus.cc:729] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda7f8046a9f469eb75ac50e4648a90c, State: Initialized, Role: FOLLOWER
I20260926 19:31:49.628275 18899 consensus_queue.cc:260] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } }
I20260926 19:31:49.628513 18899 raft_consensus.cc:388] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:49.628684 18899 raft_consensus.cc:482] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:49.628859 18899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:49.631386 18899 raft_consensus.cc:504] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } }
I20260926 19:31:49.631868 18899 leader_election.cc:302] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bda7f8046a9f469eb75ac50e4648a90c; no voters:
I20260926 19:31:49.632421 18899 leader_election.cc:288] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:31:49.632566 18904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:49.633368 18904 raft_consensus.cc:686] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [term 1 LEADER]: Becoming Leader. State: Replica: bda7f8046a9f469eb75ac50e4648a90c, State: Running, Role: LEADER
I20260926 19:31:49.633911 18904 consensus_queue.cc:237] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } }
I20260926 19:31:49.634462 18899 sys_catalog.cc:564] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:49.636957 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bda7f8046a9f469eb75ac50e4648a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } } }
I20260926 19:31:49.637223 18906 sys_catalog.cc:455] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bda7f8046a9f469eb75ac50e4648a90c. Latest consensus state: current_term: 1 leader_uuid: "bda7f8046a9f469eb75ac50e4648a90c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bda7f8046a9f469eb75ac50e4648a90c" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 46809 } } }
I20260926 19:31:49.637461 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:49.637518 18906 sys_catalog.cc:458] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:49.642748 18913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:49.647691 18913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:49.655478 18913 catalog_manager.cc:1347] Generated new cluster ID: deb0470aa54140199e010d4170cbca94
I20260926 19:31:49.655588 18913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:49.668304 18913 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:31:49.669587 18913 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:49.678515 18913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: Generated new TSK 0
I20260926 19:31:49.679283 18913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:31:49.687342 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--builtin_ntp_servers=127.1.39.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:31:50.043115 18923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:50.043390 18923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:50.043478 18923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:50.050702 18923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:50.050810 18923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:50.050916 18923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:50.060160 18923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:50.061491 18923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:50.063186 18923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:50.075915 18928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.075843 18931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.075882 18929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:50.077065 18923 server_base.cc:1034] running on GCE node
I20260926 19:31:50.077739 18923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:50.078626 18923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:50.079859 18923 hybrid_clock.cc:648] HybridClock initialized: now 1790451110079804 us; error 45 us; skew 500 ppm
I20260926 19:31:50.080168 18923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:50.082513 18923 webserver.cc:460] Webserver started at http://127.1.39.1:38475/ using document root <none> and password file <none>
I20260926 19:31:50.083182 18923 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:50.083313 18923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:50.083635 18923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:50.086457 18923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "3903c80bb896473693741e832987ecf9"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "29d67308bafbd764b6ad3126c71480bc"
server_key_iv: "8083f582494dba60b452ddbc62577bba"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.087218 18923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "3903c80bb896473693741e832987ecf9"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "29d67308bafbd764b6ad3126c71480bc"
server_key_iv: "8083f582494dba60b452ddbc62577bba"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.091995 18923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 19:31:50.095284 18937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.096505 18923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 19:31:50.096683 18923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "3903c80bb896473693741e832987ecf9"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "29d67308bafbd764b6ad3126c71480bc"
server_key_iv: "8083f582494dba60b452ddbc62577bba"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.096903 18923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:50.119881 18923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:50.120729 18923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:50.121021 18923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:50.122051 18923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:50.123533 18923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:50.123670 18923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.123791 18923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:50.123884 18923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.156483 18923 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:40113
I20260926 19:31:50.156571 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:40113 every 8 connection(s)
I20260926 19:31:50.157838 18923 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:50.158175 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 18923
I20260926 19:31:50.158335 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:31:50.158885 1180 external_mini_cluster.cc:1428] Setting key 03fc592290d1fd4e9c871b0ced3eaa96
I20260926 19:31:50.162014 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--builtin_ntp_servers=127.1.39.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:31:50.173226 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:50.160438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:38163 (local address 127.1.39.62:46809)
0926 19:31:50.161029 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:50.161035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:50.162340 (+ 1305us) server_negotiation.cc:407] Connection header received
0926 19:31:50.165197 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:50.165201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:50.165263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:50.165407 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:50.166768 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.167669 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.170178 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.170569 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.170665 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:50.172274 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:50.172281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:50.172287 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:50.172312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:50.172378 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:50.172396 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:50.172496 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:50.172706 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 19:31:50.172816 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":360,"threads_started":1}
I20260926 19:31:50.174033 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:50.161074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46809 (local address 127.1.39.1:38163)
0926 19:31:50.162183 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:50.162233 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:50.164898 (+ 2665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:50.165638 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:50.165652 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:50.165798 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:50.166584 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:50.166609 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.167816 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.167824 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:50.170035 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:50.170063 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.170792 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.170801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:50.171144 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:50.171877 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:50.171884 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:50.171936 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:50.172059 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:50.172501 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:50.172510 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:50.172516 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 19:31:50.172667 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 19:31:50.172944 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260926 19:31:50.175526 19050 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46809
I20260926 19:31:50.175876 19050 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:50.176774 19050 heartbeater.cc:511] Master 127.1.39.62:46809 requested a full tablet report, sending...
I20260926 19:31:50.179369 18864 ts_manager.cc:194] Registered new tserver with Master: 3903c80bb896473693741e832987ecf9 (127.1.39.1:40113)
I20260926 19:31:50.181653 18864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.1:38163
W20260926 19:31:50.491533 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:50.491883 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:50.492057 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:50.499761 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:50.499846 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:50.500042 19054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:50.509255 19054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:50.510577 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:50.512318 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:50.525321 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.525264 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.525362 19062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:50.525894 19054 server_base.cc:1034] running on GCE node
I20260926 19:31:50.526525 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:50.527400 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:50.528620 19054 hybrid_clock.cc:648] HybridClock initialized: now 1790451110528534 us; error 52 us; skew 500 ppm
I20260926 19:31:50.528942 19054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:50.531177 19054 webserver.cc:460] Webserver started at http://127.1.39.2:34863/ using document root <none> and password file <none>
I20260926 19:31:50.531853 19054 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:50.531988 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:50.532310 19054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:50.535182 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "50be26c235c94a46a7d1f3a5c90ecfec"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "ba92f4cebfa09c40885494998d06ff06"
server_key_iv: "d6ecc94be30dc76930314989ff965ef6"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.535982 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "50be26c235c94a46a7d1f3a5c90ecfec"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "ba92f4cebfa09c40885494998d06ff06"
server_key_iv: "d6ecc94be30dc76930314989ff965ef6"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.540781 19054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260926 19:31:50.544174 19068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.545436 19054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:50.545598 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "50be26c235c94a46a7d1f3a5c90ecfec"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "ba92f4cebfa09c40885494998d06ff06"
server_key_iv: "d6ecc94be30dc76930314989ff965ef6"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.545804 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:50.568876 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:50.569893 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:50.570185 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:50.571139 19054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:50.572608 19054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:50.572728 19054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.572860 19054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:50.572950 19054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:50.608853 19054 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:38165
I20260926 19:31:50.608912 19180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:38165 every 8 connection(s)
I20260926 19:31:50.610136 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:50.612875 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19054
I20260926 19:31:50.613067 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:31:50.613590 1180 external_mini_cluster.cc:1428] Setting key 90b8dee4958ab66aa27ebeb3a72cd52c
I20260926 19:31:50.616804 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--builtin_ntp_servers=127.1.39.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:31:50.624475 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:50.612716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:35601 (local address 127.1.39.62:46809)
0926 19:31:50.612930 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:50.612938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:50.614635 (+ 1697us) server_negotiation.cc:407] Connection header received
0926 19:31:50.616973 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:50.616979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:50.617062 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:50.617157 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:50.618563 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.619252 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.621735 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.622138 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.622238 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:50.623781 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:50.623786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:50.623789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:50.623807 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:50.623836 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:50.623847 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:50.623913 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:50.624173 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 19:31:50.624253 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:31:50.625942 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:50.613335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46809 (local address 127.1.39.2:35601)
0926 19:31:50.614445 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:50.614519 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:50.616656 (+ 2137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:50.617430 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:50.617443 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:50.617565 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:50.618363 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:50.618389 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.619405 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.619414 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:50.621572 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:50.621589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:50.622357 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:50.622366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:50.622689 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:50.623421 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:50.623427 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:50.623480 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:50.623590 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:50.623939 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:50.623951 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:50.623961 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 19:31:50.624142 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 19:31:50.624554 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":333,"threads_started":1}
I20260926 19:31:50.627380 19181 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46809
I20260926 19:31:50.627740 19181 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:50.628623 19181 heartbeater.cc:511] Master 127.1.39.62:46809 requested a full tablet report, sending...
I20260926 19:31:50.630157 18864 ts_manager.cc:194] Registered new tserver with Master: 50be26c235c94a46a7d1f3a5c90ecfec (127.1.39.2:38165)
I20260926 19:31:50.630975 18864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.2:35601
W20260926 19:31:50.946779 19184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:50.947057 19184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:50.947211 19184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:50.954725 19184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:50.954829 19184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:50.955020 19184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:50.964203 19184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:46809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:50.965530 19184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:50.967402 19184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:50.980533 19189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.980464 19192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:50.980509 19190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:50.981240 19184 server_base.cc:1034] running on GCE node
I20260926 19:31:50.981860 19184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:50.982807 19184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:50.984025 19184 hybrid_clock.cc:648] HybridClock initialized: now 1790451110983988 us; error 50 us; skew 500 ppm
I20260926 19:31:50.984336 19184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:50.986533 19184 webserver.cc:460] Webserver started at http://127.1.39.3:39033/ using document root <none> and password file <none>
I20260926 19:31:50.987221 19184 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:50.987350 19184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:50.987670 19184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:50.990487 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "2ee419fe9fd7480c88962e689edc0ecf"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "d3c49e43e159b484bc104d84d7c7457e"
server_key_iv: "64a66f5d62b1e89b92999916566d7c8a"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.991250 19184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "2ee419fe9fd7480c88962e689edc0ecf"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "d3c49e43e159b484bc104d84d7c7457e"
server_key_iv: "64a66f5d62b1e89b92999916566d7c8a"
server_key_version: "encryptionkey@0"
I20260926 19:31:50.996179 19184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:31:50.999536 19198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:51.000959 19184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 19:31:51.001132 19184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "2ee419fe9fd7480c88962e689edc0ecf"
format_stamp: "Formatted at 2026-09-26 19:31:50 on dist-test-slave-9wb0"
server_key: "d3c49e43e159b484bc104d84d7c7457e"
server_key_iv: "64a66f5d62b1e89b92999916566d7c8a"
server_key_version: "encryptionkey@0"
I20260926 19:31:51.001380 19184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:51.029947 19184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:51.030804 19184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:51.031082 19184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:51.032114 19184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:51.033620 19184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:51.033740 19184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:51.033876 19184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:51.033967 19184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:51.067787 19184 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:35209
I20260926 19:31:51.067842 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:35209 every 8 connection(s)
I20260926 19:31:51.069046 19184 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:51.076347 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19184
I20260926 19:31:51.076524 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:31:51.077059 1180 external_mini_cluster.cc:1428] Setting key f9eeb469cb739eae963a67aefded6f54
I20260926 19:31:51.081689 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.071530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:40411 (local address 127.1.39.62:46809)
0926 19:31:51.071701 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.071705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.073096 (+ 1391us) server_negotiation.cc:407] Connection header received
0926 19:31:51.074825 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.074829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.074895 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.074984 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:51.075933 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.077118 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.078892 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.079392 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.079544 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.080855 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:51.080862 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:51.080867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:51.080895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:51.080926 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:51.080938 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:51.081033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.081248 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.081364 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:31:51.082649 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.072108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:46809 (local address 127.1.39.3:40411)
0926 19:31:51.072943 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.072976 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.074592 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.075177 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.075184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.075265 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:51.075776 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.075790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.077250 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.077254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.078756 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.078767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.079622 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.079631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.079855 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.080557 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:51.080559 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:51.080595 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:51.080667 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:51.081032 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:51.081044 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:51.081054 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.081239 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.081620 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":265,"threads_started":1}
I20260926 19:31:51.083913 19311 heartbeater.cc:347] Connected to a master server at 127.1.39.62:46809
I20260926 19:31:51.084201 19311 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:51.084869 19311 heartbeater.cc:511] Master 127.1.39.62:46809 requested a full tablet report, sending...
I20260926 19:31:51.086226 18864 ts_manager.cc:194] Registered new tserver with Master: 2ee419fe9fd7480c88962e689edc0ecf (127.1.39.3:35209)
I20260926 19:31:51.087025 18864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.3:40411
I20260926 19:31:51.092298 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:31:51.101742 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.096180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.1.39.62:46809)
0926 19:31:51.096400 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.096406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.096696 (+ 290us) server_negotiation.cc:407] Connection header received
0926 19:31:51.096824 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.096827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.096893 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.096971 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:51.097452 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.098127 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.099381 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.099779 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.099866 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.100894 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:51.100898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:51.100901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:51.100917 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:51.100947 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:51.100958 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:51.101029 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.101269 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.101390 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:31:51.105708 18864 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:31:51.112473 18864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:31:51.135673 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.125531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:38165 (local address 127.0.0.1:37946)
0926 19:31:51.126676 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.126713 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.126938 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.127616 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.127622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.127649 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:51.128046 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.128063 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130156 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.130164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.133126 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.133140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.134662 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.134666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.134856 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.134887 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.135034 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.135229 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"mutex_wait_us":263,"spinlock_wait_cycles":1408,"thread_start_us":744,"threads_started":1}
I20260926 19:31:51.135905 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.124170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.1.39.3:35209)
0926 19:31:51.124945 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.124956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.127077 (+ 2121us) server_negotiation.cc:407] Connection header received
0926 19:31:51.127265 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.127271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.127436 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.127619 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:51.128772 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130323 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.132533 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.134335 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.134438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.134752 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.135380 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.135534 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":431,"threads_started":1}
I20260926 19:31:51.136335 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.124628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.1.39.1:40113)
0926 19:31:51.125381 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.125391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.126834 (+ 1443us) server_negotiation.cc:407] Connection header received
0926 19:31:51.127202 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.127211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.127436 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.127612 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:51.128590 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130236 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.133423 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.134879 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.135019 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.135212 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.135766 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.135986 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":577,"threads_started":1}
I20260926 19:31:51.136610 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.125229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.1.39.2:38165)
0926 19:31:51.125924 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.125934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.126900 (+ 966us) server_negotiation.cc:407] Connection header received
0926 19:31:51.127202 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.127211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.127407 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.127589 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:51.128262 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130025 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.133374 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.134297 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.134383 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.134545 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.136111 (+ 1566us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.136270 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":396,"threads_started":1}
I20260926 19:31:51.135860 19323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.123912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35209 (local address 127.0.0.1:59434)
0926 19:31:51.126957 (+ 3045us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.126978 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.127092 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.127894 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.127908 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.127930 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:51.128568 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.128582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130476 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.130480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.132282 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.132299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.134928 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.134937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.135139 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.135179 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.135332 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.135457 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"mutex_wait_us":192,"thread_start_us":2259,"threads_started":1}
I20260926 19:31:51.136818 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.124133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:46982)
0926 19:31:51.126690 (+ 2557us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.126715 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.126938 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.127848 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.127852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.127871 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:51.128426 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.128435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.130378 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.130383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.133126 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.133140 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.135075 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.135080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.135276 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.135303 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.135386 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.135509 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2313,"mutex_wait_us":199,"spinlock_wait_cycles":9856,"thread_start_us":1563,"threads_started":1}
I20260926 19:31:51.140251 18985 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e (DEFAULT_TABLE table=test-table [id=ef436f77bf954c0d844c8cace52a3813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:51.140503 19246 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e (DEFAULT_TABLE table=test-table [id=ef436f77bf954c0d844c8cace52a3813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:51.140503 19116 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e (DEFAULT_TABLE table=test-table [id=ef436f77bf954c0d844c8cace52a3813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:51.141626 18985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6defbb3e9749cfb418b5bf55b9142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:51.141850 19246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6defbb3e9749cfb418b5bf55b9142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:51.142156 19116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6defbb3e9749cfb418b5bf55b9142e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:51.152881 19329 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Bootstrap starting.
I20260926 19:31:51.156450 19329 tablet_bootstrap.cc:654] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:51.157609 19329 log.cc:824] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:51.160501 19329 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: No bootstrap required, opened a new log
I20260926 19:31:51.160777 19329 ts_tablet_manager.cc:1397] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260926 19:31:51.161401 19330 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Bootstrap starting.
I20260926 19:31:51.165000 19330 tablet_bootstrap.cc:654] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:51.166298 19330 log.cc:824] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:51.170584 19332 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Bootstrap starting.
I20260926 19:31:51.173143 19329 raft_consensus.cc:348] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.173619 19329 raft_consensus.cc:374] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:51.173713 19330 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: No bootstrap required, opened a new log
I20260926 19:31:51.173727 19329 raft_consensus.cc:729] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3903c80bb896473693741e832987ecf9, State: Initialized, Role: FOLLOWER
I20260926 19:31:51.173789 19332 tablet_bootstrap.cc:654] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:51.174005 19330 ts_tablet_manager.cc:1397] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260926 19:31:51.174425 19329 consensus_queue.cc:260] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.175623 19332 log.cc:824] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:51.177974 19050 heartbeater.cc:503] Master 127.1.39.62:46809 was elected leader, sending a full tablet report...
I20260926 19:31:51.178306 19332 tablet_bootstrap.cc:492] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: No bootstrap required, opened a new log
I20260926 19:31:51.178546 19332 ts_tablet_manager.cc:1397] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260926 19:31:51.178599 19329 ts_tablet_manager.cc:1428] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260926 19:31:51.184710 19330 raft_consensus.cc:348] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.185071 19330 raft_consensus.cc:374] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:51.185189 19330 raft_consensus.cc:729] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee419fe9fd7480c88962e689edc0ecf, State: Initialized, Role: FOLLOWER
I20260926 19:31:51.185905 19330 consensus_queue.cc:260] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.186932 19311 heartbeater.cc:503] Master 127.1.39.62:46809 was elected leader, sending a full tablet report...
I20260926 19:31:51.187171 19332 raft_consensus.cc:348] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.187566 19332 raft_consensus.cc:374] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:51.187656 19330 ts_tablet_manager.cc:1428] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260926 19:31:51.187678 19332 raft_consensus.cc:729] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50be26c235c94a46a7d1f3a5c90ecfec, State: Initialized, Role: FOLLOWER
I20260926 19:31:51.188311 19332 consensus_queue.cc:260] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.189405 19181 heartbeater.cc:503] Master 127.1.39.62:46809 was elected leader, sending a full tablet report...
I20260926 19:31:51.189894 19332 ts_tablet_manager.cc:1428] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260926 19:31:51.218937 19334 raft_consensus.cc:482] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:51.219220 19334 raft_consensus.cc:504] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.220844 19334 leader_election.cc:288] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ee419fe9fd7480c88962e689edc0ecf (127.1.39.3:35209), 50be26c235c94a46a7d1f3a5c90ecfec (127.1.39.2:38165)
I20260926 19:31:51.227872 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.221376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:56649 (local address 127.1.39.3:35209)
0926 19:31:51.221575 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.221580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.222406 (+ 826us) server_negotiation.cc:407] Connection header received
0926 19:31:51.222747 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.222753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.222829 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.222917 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:51.223575 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.224285 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.226199 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.226959 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.227098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.227230 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.227504 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.227633 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:31:51.227911 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.221363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:46163 (local address 127.1.39.2:38165)
0926 19:31:51.221562 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.221570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.222711 (+ 1141us) server_negotiation.cc:407] Connection header received
0926 19:31:51.222965 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.222972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.223063 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.223165 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:51.223742 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.224409 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.226261 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.227013 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.227155 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.227249 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.227562 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.227681 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:31:51.227998 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.221352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:35209 (local address 127.1.39.1:56649)
0926 19:31:51.222280 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.222308 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.222568 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.222955 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.222959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.222980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:51.223421 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.223433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.224423 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.224431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.226012 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.226027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.227154 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.227161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.227349 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.227374 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.227477 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.227607 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":23,"thread_start_us":498,"threads_started":1}
I20260926 19:31:51.228259 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.221384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:38165 (local address 127.1.39.1:46163)
0926 19:31:51.222592 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.222619 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:51.222807 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:51.223179 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:51.223184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:51.223208 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:51.223597 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.223606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.224547 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.224554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.226055 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:51.226069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.227206 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.227212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:51.227400 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.227423 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 19:31:51.227527 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 19:31:51.227644 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":45,"thread_start_us":839,"threads_started":1}
I20260926 19:31:51.229072 19266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6defbb3e9749cfb418b5bf55b9142e" candidate_uuid: "3903c80bb896473693741e832987ecf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee419fe9fd7480c88962e689edc0ecf" is_pre_election: true
I20260926 19:31:51.229161 19136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6defbb3e9749cfb418b5bf55b9142e" candidate_uuid: "3903c80bb896473693741e832987ecf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" is_pre_election: true
I20260926 19:31:51.229629 19266 raft_consensus.cc:2454] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3903c80bb896473693741e832987ecf9 in term 0.
I20260926 19:31:51.229696 19136 raft_consensus.cc:2454] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3903c80bb896473693741e832987ecf9 in term 0.
I20260926 19:31:51.230362 18938 leader_election.cc:302] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3903c80bb896473693741e832987ecf9, 50be26c235c94a46a7d1f3a5c90ecfec; no voters:
I20260926 19:31:51.230762 19334 raft_consensus.cc:2781] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:31:51.230924 19334 raft_consensus.cc:482] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:31:51.231040 19334 raft_consensus.cc:3037] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:51.233168 19334 raft_consensus.cc:504] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.233825 19334 leader_election.cc:288] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [CANDIDATE]: Term 1 election: Requested vote from peers 2ee419fe9fd7480c88962e689edc0ecf (127.1.39.3:35209), 50be26c235c94a46a7d1f3a5c90ecfec (127.1.39.2:38165)
I20260926 19:31:51.234208 19266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6defbb3e9749cfb418b5bf55b9142e" candidate_uuid: "3903c80bb896473693741e832987ecf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ee419fe9fd7480c88962e689edc0ecf"
I20260926 19:31:51.234241 19136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6defbb3e9749cfb418b5bf55b9142e" candidate_uuid: "3903c80bb896473693741e832987ecf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50be26c235c94a46a7d1f3a5c90ecfec"
I20260926 19:31:51.234413 19266 raft_consensus.cc:3037] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:51.234427 19136 raft_consensus.cc:3037] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:51.236660 19266 raft_consensus.cc:2454] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3903c80bb896473693741e832987ecf9 in term 1.
I20260926 19:31:51.236685 19136 raft_consensus.cc:2454] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3903c80bb896473693741e832987ecf9 in term 1.
I20260926 19:31:51.237092 18940 leader_election.cc:302] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ee419fe9fd7480c88962e689edc0ecf, 3903c80bb896473693741e832987ecf9; no voters:
I20260926 19:31:51.237403 19334 raft_consensus.cc:2781] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:51.237845 19334 raft_consensus.cc:686] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3903c80bb896473693741e832987ecf9, State: Running, Role: LEADER
I20260926 19:31:51.238305 19334 consensus_queue.cc:237] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } }
I20260926 19:31:51.243037 18863 catalog_manager.cc:5495] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3903c80bb896473693741e832987ecf9 (127.1.39.1). New cstate: current_term: 1 leader_uuid: "3903c80bb896473693741e832987ecf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3903c80bb896473693741e832987ecf9" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 } health_report { overall_health: UNKNOWN } } }
I20260926 19:31:51.257934 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.253067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.1.39.1:40113)
0926 19:31:51.253239 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.253244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.253355 (+ 111us) server_negotiation.cc:407] Connection header received
0926 19:31:51.253544 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.253548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.253661 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.253781 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:51.254051 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.254700 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.256094 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.256472 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.256568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.256969 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:51.257491 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:31:51.257616 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.257675 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.257731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:31:51.266959 19266 raft_consensus.cc:1261] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 1 FOLLOWER]: Refusing update from remote peer 3903c80bb896473693741e832987ecf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:51.266999 19136 raft_consensus.cc:1261] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 1 FOLLOWER]: Refusing update from remote peer 3903c80bb896473693741e832987ecf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:51.267992 19340 consensus_queue.cc:1035] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50be26c235c94a46a7d1f3a5c90ecfec" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 38165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:51.268368 19334 consensus_queue.cc:1035] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ee419fe9fd7480c88962e689edc0ecf" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 35209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:51.293598 18863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:51.293895 18863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:51.297283 18863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bda7f8046a9f469eb75ac50e4648a90c: Sending DeleteTablet for 3 replicas of tablet 8a6defbb3e9749cfb418b5bf55b9142e
I20260926 19:31:51.298496 19246 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:51 UTC) from {username='slave'} at 127.0.0.1:59434
I20260926 19:31:51.298506 18985 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:51 UTC) from {username='slave'} at 127.0.0.1:46982
I20260926 19:31:51.298799 19116 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6defbb3e9749cfb418b5bf55b9142e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:51 UTC) from {username='slave'} at 127.0.0.1:37946
I20260926 19:31:51.301048 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18923
I20260926 19:31:51.311427 19352 tablet_replica.cc:330] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: stopping tablet replica
I20260926 19:31:51.312958 19352 raft_consensus.cc:2229] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:51.314493 19352 raft_consensus.cc:2258] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:51.318040 19352 ts_tablet_manager.cc:1905] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:51.319262 19353 tablet_replica.cc:330] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: stopping tablet replica
I20260926 19:31:51.319983 19353 raft_consensus.cc:2229] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:31:51.321022 19353 raft_consensus.cc:2258] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:51.324919 19354 tablet_replica.cc:330] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: stopping tablet replica
I20260926 19:31:51.325706 19354 raft_consensus.cc:2229] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:51.326260 19354 raft_consensus.cc:2258] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:51.327126 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.305610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.1.39.1:40113)
0926 19:31:51.305792 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.305799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.317669 (+ 11870us) server_negotiation.cc:407] Connection header received
0926 19:31:51.318180 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.318186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.318296 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.318385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:51.319105 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.320374 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.322859 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.323528 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.323648 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.325391 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:51.325400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:51.325416 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:51.326197 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:51.326257 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:51.326280 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:51.326391 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.326622 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.326728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:31:51.328235 19045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:51.329206 19354 ts_tablet_manager.cc:1905] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:51.331604 19352 ts_tablet_manager.cc:1918] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:51.331839 19352 log.cc:1196] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/8a6defbb3e9749cfb418b5bf55b9142e
I20260926 19:31:51.332975 19352 ts_tablet_manager.cc:1939] T 8a6defbb3e9749cfb418b5bf55b9142e P 2ee419fe9fd7480c88962e689edc0ecf: Deleting consensus metadata
I20260926 19:31:51.334981 18850 catalog_manager.cc:4841] TS 2ee419fe9fd7480c88962e689edc0ecf (127.1.39.3:35209): tablet 8a6defbb3e9749cfb418b5bf55b9142e (table test-table [id=ef436f77bf954c0d844c8cace52a3813]) successfully deleted
I20260926 19:31:51.336444 19354 ts_tablet_manager.cc:1918] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:51.336628 19354 log.cc:1196] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/8a6defbb3e9749cfb418b5bf55b9142e
I20260926 19:31:51.337067 19354 ts_tablet_manager.cc:1939] T 8a6defbb3e9749cfb418b5bf55b9142e P 50be26c235c94a46a7d1f3a5c90ecfec: Deleting consensus metadata
I20260926 19:31:51.338572 18848 catalog_manager.cc:4841] TS 50be26c235c94a46a7d1f3a5c90ecfec (127.1.39.2:38165): tablet 8a6defbb3e9749cfb418b5bf55b9142e (table test-table [id=ef436f77bf954c0d844c8cace52a3813]) successfully deleted
I20260926 19:31:51.329084 19353 ts_tablet_manager.cc:1905] T 8a6defbb3e9749cfb418b5bf55b9142e P 3903c80bb896473693741e832987ecf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:51.425652 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18923
W20260926 19:31:51.444555 18851 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.39.1:40113 (error 108)
I20260926 19:31:51.445256 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19054
I20260926 19:31:51.446600 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.446040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59702)
0926 19:31:51.446298 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.446419 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260926 19:31:51.446859 18851 catalog_manager.cc:4584] TS 3903c80bb896473693741e832987ecf9 (127.1.39.1:40113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a6defbb3e9749cfb418b5bf55b9142e: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
I20260926 19:31:51.452245 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.446184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.1.39.2:38165)
0926 19:31:51.446350 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.446359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.446380 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:31:51.446479 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.446485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.446588 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.446714 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:51.447154 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.447893 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.449334 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.449749 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.449878 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.450932 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:51.450941 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:51.450954 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:51.451509 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:51.451555 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:51.451573 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:51.451704 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.451835 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.451940 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:31:51.452893 19176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:51.500581 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.500013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59718)
0926 19:31:51.500247 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.500356 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 19:31:51.501724 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.501285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59720)
0926 19:31:51.501493 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.501591 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 19:31:51.541584 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19054
I20260926 19:31:51.559319 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19184
I20260926 19:31:51.562016 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.561601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59736)
0926 19:31:51.561777 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.561851 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 19:31:51.563086 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.562737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59742)
0926 19:31:51.562912 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.562985 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 19:31:51.566715 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.560086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.1.39.3:35209)
0926 19:31:51.560285 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:51.560293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:51.560337 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:31:51.560560 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:51.560567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:51.560676 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:51.560794 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:51.561212 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.562011 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.563657 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:51.564074 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:51.564207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:51.565221 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:51.565231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:51.565246 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:51.565904 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:51.566002 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:51.566028 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:51.566136 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:51.566283 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 19:31:51.566391 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:31:51.567420 19306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:51.643656 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.643220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59758)
0926 19:31:51.643407 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.643502 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 19:31:51.644708 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:51.644326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40113 (local address 127.0.0.1:59768)
0926 19:31:51.644519 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:51.644600 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 19:31:51.656651 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19184
I20260926 19:31:51.673674 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 18833
I20260926 19:31:51.674721 18894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:51.784606 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 18833
2026-09-26T19:31:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2683 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:31:51 dist-test-slave-9wb0 krb5kdc[19363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:31:51 dist-test-slave-9wb0 krb5kdc[19363](info): set up 2 sockets
Sep 26 19:31:51 dist-test-slave-9wb0 krb5kdc[19363](info): commencing operation
krb5kdc: starting...
W20260926 19:31:52.905526 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:31:53 dist-test-slave-9wb0 krb5kdc[19363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:31:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:31:53.176884 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40767
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:53.507952 19376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:53.508236 19376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:53.508394 19376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:53.515632 19376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:53.515734 19376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:53.515841 19376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:53.515941 19376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:53.516024 19376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:53.525758 19376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:40767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:53.527086 19376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:53.528364 19376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:53.540421 19384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:53.540522 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:53.540522 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:53.541211 19376 server_base.cc:1034] running on GCE node
I20260926 19:31:53.541980 19376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:53.543309 19376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:53.544529 19376 hybrid_clock.cc:648] HybridClock initialized: now 1790451113544481 us; error 37 us; skew 500 ppm
I20260926 19:31:53.544871 19376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:53 dist-test-slave-9wb0 krb5kdc[19363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:53.548024 19376 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:31:53.549526 19376 webserver.cc:460] Webserver started at http://127.1.39.62:34763/ using document root <none> and password file <none>
I20260926 19:31:53.550179 19376 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:53.550338 19376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:53.550642 19376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:53.553541 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "96b3ba254b6749078011b99f962f18d9"
format_stamp: "Formatted at 2026-09-26 19:31:53 on dist-test-slave-9wb0"
server_key: "2486fe6af03823edda46a39becd0d12f"
server_key_iv: "0f303f9ec55fb92ef3fe0ff523328f00"
server_key_version: "encryptionkey@0"
I20260926 19:31:53.554250 19376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "96b3ba254b6749078011b99f962f18d9"
format_stamp: "Formatted at 2026-09-26 19:31:53 on dist-test-slave-9wb0"
server_key: "2486fe6af03823edda46a39becd0d12f"
server_key_iv: "0f303f9ec55fb92ef3fe0ff523328f00"
server_key_version: "encryptionkey@0"
I20260926 19:31:53.559162 19376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 19:31:53.562646 19391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:53.564157 19376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:31:53.564366 19376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "96b3ba254b6749078011b99f962f18d9"
format_stamp: "Formatted at 2026-09-26 19:31:53 on dist-test-slave-9wb0"
server_key: "2486fe6af03823edda46a39becd0d12f"
server_key_iv: "0f303f9ec55fb92ef3fe0ff523328f00"
server_key_version: "encryptionkey@0"
I20260926 19:31:53.564615 19376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:53.586089 19376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:53.590494 19376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:53.590790 19376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:53.609598 19376 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:40767
I20260926 19:31:53.609679 19442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:40767 every 8 connection(s)
I20260926 19:31:53.610936 19376 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:53.612398 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19376
I20260926 19:31:53.612628 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:31:53.613193 1180 external_mini_cluster.cc:1428] Setting key 0eacd440da1209c7f06c89b1c6fafb05
I20260926 19:31:53.615666 19443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:53.626619 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: Bootstrap starting.
Sep 26 19:31:53 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:53.630111 19443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:53.631269 19443 log.cc:824] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:53.634593 19443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: No bootstrap required, opened a new log
I20260926 19:31:53.635164 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:53.615270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.1.39.62:40767)
0926 19:31:53.616087 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:53.616099 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:53.616136 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:31:53.617875 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:53.617894 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:53.618266 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:53.618659 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:53.620196 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:53.621545 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:53.622827 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:53.623494 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:53.623725 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:53.627734 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:53.627741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:53.627754 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:53.627789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:53.630931 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:53.631709 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:53.631719 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:53.631729 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:53.631966 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:53.632417 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:53.632424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:53.632427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:53.632941 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:53.633167 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:53.633791 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 19:31:53.634106 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":375,"threads_started":1}
I20260926 19:31:53.642616 19443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } }
I20260926 19:31:53.643047 19443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:53.643167 19443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b3ba254b6749078011b99f962f18d9, State: Initialized, Role: FOLLOWER
I20260926 19:31:53.643736 19443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } }
I20260926 19:31:53.643998 19443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:53.644167 19443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:53.644351 19443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:53.646788 19443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } }
I20260926 19:31:53.647399 19443 leader_election.cc:302] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b3ba254b6749078011b99f962f18d9; no voters:
I20260926 19:31:53.648161 19443 leader_election.cc:288] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:31:53.648300 19448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:53.649092 19448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [term 1 LEADER]: Becoming Leader. State: Replica: 96b3ba254b6749078011b99f962f18d9, State: Running, Role: LEADER
I20260926 19:31:53.649781 19448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } }
I20260926 19:31:53.650676 19443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:53.652871 19450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b3ba254b6749078011b99f962f18d9. Latest consensus state: current_term: 1 leader_uuid: "96b3ba254b6749078011b99f962f18d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } } }
I20260926 19:31:53.653534 19450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:53.653278 19449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b3ba254b6749078011b99f962f18d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3ba254b6749078011b99f962f18d9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 40767 } } }
I20260926 19:31:53.653724 19449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:53.655952 19455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:53.660531 19455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:53.669803 19455 catalog_manager.cc:1347] Generated new cluster ID: f3996e44948a47ccafb4aad7da66243f
I20260926 19:31:53.669922 19455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:53.715939 19455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:31:53.717378 19455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:53.724339 19455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: Generated new TSK 0
I20260926 19:31:53.725193 19455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:31:53.804638 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:54.170589 19469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:54.170849 19469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:54.170941 19469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:54.178210 19469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:54.178313 19469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:54.178423 19469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:54.188218 19469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:54.189529 19469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:54.190806 19469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:54.203449 19474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:54.203586 19475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:54.204022 19477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:54.204573 19469 server_base.cc:1034] running on GCE node
I20260926 19:31:54.205226 19469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:54.206151 19469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:54.207350 19469 hybrid_clock.cc:648] HybridClock initialized: now 1790451114207304 us; error 42 us; skew 500 ppm
I20260926 19:31:54.207621 19469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:54 dist-test-slave-9wb0 krb5kdc[19363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451114, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:54.210886 19469 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:31:54.212232 19469 webserver.cc:460] Webserver started at http://127.1.39.1:37807/ using document root <none> and password file <none>
I20260926 19:31:54.212849 19469 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:54.212945 19469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:54.213191 19469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:54.216171 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "c95c4d95deb34945955152789ca14993"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "f8d2bbb47cbb142842a0c8fb8d988d31"
server_key_iv: "98c05b201c2106cf96ac9c2e52a8534d"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.216910 19469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "c95c4d95deb34945955152789ca14993"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "f8d2bbb47cbb142842a0c8fb8d988d31"
server_key_iv: "98c05b201c2106cf96ac9c2e52a8534d"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.221967 19469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 19:31:54.225555 19484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.226809 19469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 19:31:54.226984 19469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "c95c4d95deb34945955152789ca14993"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "f8d2bbb47cbb142842a0c8fb8d988d31"
server_key_iv: "98c05b201c2106cf96ac9c2e52a8534d"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.227183 19469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:54.245759 19469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:54.249697 19469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:54.249998 19469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:54.251071 19469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:54.252506 19469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:54.252579 19469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.252700 19469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:54.252767 19469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.287776 19469 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:46869
I20260926 19:31:54.287861 19596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:46869 every 8 connection(s)
I20260926 19:31:54.289081 19469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:54.295074 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19469
I20260926 19:31:54.295266 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:31:54.295928 1180 external_mini_cluster.cc:1428] Setting key d2f8919e56913e02688ae2d1a7b2a71b
Sep 26 19:31:54 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451114, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:54.313474 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:54.291773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:47055 (local address 127.1.39.62:40767)
0926 19:31:54.292270 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:54.292275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:54.293100 (+ 825us) server_negotiation.cc:407] Connection header received
0926 19:31:54.295324 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:54.295333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:54.295429 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:54.295553 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:54.297827 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.298724 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.301414 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.301811 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.301916 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:54.306539 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:54.306545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:54.306548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:54.306581 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:54.309778 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:54.310607 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:54.310615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:54.310622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:54.310778 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:54.311315 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:54.311323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:54.311330 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:54.311707 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:54.311854 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:54.312687 (+ 833us) server_negotiation.cc:299] Negotiation successful
0926 19:31:54.312943 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":280,"threads_started":1}
I20260926 19:31:54.314343 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:54.292219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40767 (local address 127.1.39.1:47055)
0926 19:31:54.292930 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:54.292980 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:54.294899 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:54.295853 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:54.295865 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:54.296810 (+ 945us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:54.297648 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:54.297669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.298898 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.298904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:54.301232 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:54.301251 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.302060 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.302069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:54.302429 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:54.303197 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:54.303238 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:54.306295 (+ 3057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:54.309910 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:54.309917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:54.309930 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:54.310454 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:54.310929 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:54.310934 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:54.310936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:54.311181 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:54.311855 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:54.311862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:54.312220 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 19:31:54.312532 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 19:31:54.312972 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":295,"threads_started":1}
I20260926 19:31:54.316190 19597 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40767
I20260926 19:31:54.316547 19597 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:54.317477 19597 heartbeater.cc:511] Master 127.1.39.62:40767 requested a full tablet report, sending...
I20260926 19:31:54.319733 19408 ts_manager.cc:194] Registered new tserver with Master: c95c4d95deb34945955152789ca14993 (127.1.39.1:46869)
I20260926 19:31:54.321357 19408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:47055
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:31:54.360837 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:54.696528 19603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:54.696838 19603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:54.697007 19603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:54.704236 19603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:54.704387 19603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:54.704577 19603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:54.714452 19603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:54.715749 19603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:54.717049 19603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:54.730436 19609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:54.730406 19608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:54.730685 19611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:54.731443 19603 server_base.cc:1034] running on GCE node
I20260926 19:31:54.732110 19603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:54.733093 19603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:54.734292 19603 hybrid_clock.cc:648] HybridClock initialized: now 1790451114734220 us; error 61 us; skew 500 ppm
I20260926 19:31:54.734676 19603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:54 dist-test-slave-9wb0 krb5kdc[19363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451114, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:54.742815 19603 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:31:54.744381 19603 webserver.cc:460] Webserver started at http://127.1.39.2:34181/ using document root <none> and password file <none>
I20260926 19:31:54.745086 19603 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:54.745230 19603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:54.745577 19603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:54.748461 19603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "9484f32eacbe41599419c7cda48dfa08"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "eadac86c539e8d5581466121af152da4"
server_key_iv: "880a3596199dc7453a3f0b2976e9ae57"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.749249 19603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "9484f32eacbe41599419c7cda48dfa08"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "eadac86c539e8d5581466121af152da4"
server_key_iv: "880a3596199dc7453a3f0b2976e9ae57"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.754488 19603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 19:31:54.757869 19618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.759172 19603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:31:54.759346 19603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "9484f32eacbe41599419c7cda48dfa08"
format_stamp: "Formatted at 2026-09-26 19:31:54 on dist-test-slave-9wb0"
server_key: "eadac86c539e8d5581466121af152da4"
server_key_iv: "880a3596199dc7453a3f0b2976e9ae57"
server_key_version: "encryptionkey@0"
I20260926 19:31:54.759541 19603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:54.782646 19603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:54.787060 19603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:54.787374 19603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:54.788378 19603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:54.789822 19603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:54.789927 19603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.790081 19603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:54.790155 19603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:54.823343 19603 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:41653
I20260926 19:31:54.823376 19730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:41653 every 8 connection(s)
I20260926 19:31:54.824618 19603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:54.827172 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19603
I20260926 19:31:54.827391 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:31:54.827878 1180 external_mini_cluster.cc:1428] Setting key c0f0e24679b4a77fab6c4b0b853f078e
Sep 26 19:31:54 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451114, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:54.845834 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:54.827132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:40367 (local address 127.1.39.62:40767)
0926 19:31:54.827678 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:54.827685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:54.828674 (+ 989us) server_negotiation.cc:407] Connection header received
0926 19:31:54.830636 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:54.830644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:54.830728 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:54.830865 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:54.832582 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.833770 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.835245 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.835991 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.836138 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:54.839840 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:54.839846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:54.839849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:54.839880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:54.842554 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:54.843454 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:54.843458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:54.843460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:54.843611 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:54.844081 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:54.844084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:54.844086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:54.844328 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:54.844434 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:54.845411 (+ 977us) server_negotiation.cc:299] Negotiation successful
0926 19:31:54.845572 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":293,"threads_started":1}
I20260926 19:31:54.847282 19734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:54.827693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40767 (local address 127.1.39.2:40367)
0926 19:31:54.828522 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:54.828566 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:54.830372 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:54.831148 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:54.831156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:54.831687 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:54.832408 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:54.832428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.833909 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.833913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:54.835106 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:54.835117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:54.836197 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:54.836202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:54.836410 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:54.837092 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:54.837114 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:54.839597 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:54.842686 (+ 3089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:54.842698 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:54.842717 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:54.843342 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:54.843721 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:54.843729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:54.843736 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:54.843972 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:54.844456 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:54.844466 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:54.844912 (+ 446us) client_negotiation.cc:769] Sending connection context
0926 19:31:54.845262 (+ 350us) client_negotiation.cc:240] Negotiation successful
0926 19:31:54.845826 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":347,"threads_started":1}
I20260926 19:31:54.848701 19731 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40767
I20260926 19:31:54.849005 19731 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:54.849763 19731 heartbeater.cc:511] Master 127.1.39.62:40767 requested a full tablet report, sending...
I20260926 19:31:54.851333 19408 ts_manager.cc:194] Registered new tserver with Master: 9484f32eacbe41599419c7cda48dfa08 (127.1.39.2:41653)
I20260926 19:31:54.852385 19408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:40367
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:31:54.892701 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:42329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:31:55.217553 19737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:55.217875 19737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:55.218040 19737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:55.225157 19737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:55.225342 19737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:55.225530 19737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:55.235386 19737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:42329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:40767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:55.236702 19737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:55.238082 19737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:55.251299 19742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:55.251547 19743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:55.251883 19745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:55.252619 19737 server_base.cc:1034] running on GCE node
I20260926 19:31:55.253216 19737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:55.254185 19737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:55.255406 19737 hybrid_clock.cc:648] HybridClock initialized: now 1790451115255348 us; error 57 us; skew 500 ppm
I20260926 19:31:55.255791 19737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:31:55 dist-test-slave-9wb0 krb5kdc[19363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:31:55.258949 19737 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:31:55.260445 19737 webserver.cc:460] Webserver started at http://127.1.39.3:45987/ using document root <none> and password file <none>
I20260926 19:31:55.261116 19737 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:55.261256 19737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:55.261603 19737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:55.264628 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "6817fe4aafca47b4b2c55d777cea52ee"
format_stamp: "Formatted at 2026-09-26 19:31:55 on dist-test-slave-9wb0"
server_key: "e21cee12b51cb0e96c0d05bd7b647a0c"
server_key_iv: "4fb1cf8a4657e9c368c61a6849cc16ae"
server_key_version: "encryptionkey@0"
I20260926 19:31:55.265466 19737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "6817fe4aafca47b4b2c55d777cea52ee"
format_stamp: "Formatted at 2026-09-26 19:31:55 on dist-test-slave-9wb0"
server_key: "e21cee12b51cb0e96c0d05bd7b647a0c"
server_key_iv: "4fb1cf8a4657e9c368c61a6849cc16ae"
server_key_version: "encryptionkey@0"
I20260926 19:31:55.270998 19737 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:31:55.274641 19752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:55.276005 19737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:31:55.276189 19737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "6817fe4aafca47b4b2c55d777cea52ee"
format_stamp: "Formatted at 2026-09-26 19:31:55 on dist-test-slave-9wb0"
server_key: "e21cee12b51cb0e96c0d05bd7b647a0c"
server_key_iv: "4fb1cf8a4657e9c368c61a6849cc16ae"
server_key_version: "encryptionkey@0"
I20260926 19:31:55.276401 19737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:55.288559 19737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:55.292920 19737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:55.293221 19737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:55.294164 19737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:55.295486 19737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:55.295601 19737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:55.295751 19737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:55.295825 19737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:55.324194 19597 heartbeater.cc:503] Master 127.1.39.62:40767 was elected leader, sending a full tablet report...
I20260926 19:31:55.330607 19737 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:33435
I20260926 19:31:55.330641 19864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:33435 every 8 connection(s)
I20260926 19:31:55.331969 19737 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:55.338868 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19737
I20260926 19:31:55.339061 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:31:55.339591 1180 external_mini_cluster.cc:1428] Setting key c836c4389f369ac346272f97514e5026
Sep 26 19:31:55 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:31:55.352094 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.334573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:53727 (local address 127.1.39.62:40767)
0926 19:31:55.334746 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.334750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.335831 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 19:31:55.337723 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.337730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.337820 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.337931 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:55.339820 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.340612 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.342594 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.342971 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.343070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.346505 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:55.346510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:55.346513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:55.346548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:55.349286 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.350021 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.350024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.350027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.350142 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.350527 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.350530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.350532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.350857 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:55.350965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.351623 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.351843 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:31:55.353132 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.335009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:40767 (local address 127.1.39.3:53727)
0926 19:31:55.335680 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.335723 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.337447 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.338157 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.338166 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.338797 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:55.339588 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.339616 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.340778 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.340786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.342451 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.342475 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.343160 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.343165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.343344 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.343945 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:31:55.343967 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:55.346292 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:55.349481 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:55.349488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:55.349500 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:55.349914 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:55.350237 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:31:55.350240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:31:55.350242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:31:55.350402 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:31:55.350973 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:55.350979 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:55.351277 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.351494 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.351957 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":270,"threads_started":1}
I20260926 19:31:55.354642 19865 heartbeater.cc:347] Connected to a master server at 127.1.39.62:40767
I20260926 19:31:55.354979 19865 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:55.355675 19865 heartbeater.cc:511] Master 127.1.39.62:40767 requested a full tablet report, sending...
I20260926 19:31:55.357092 19408 ts_manager.cc:194] Registered new tserver with Master: 6817fe4aafca47b4b2c55d777cea52ee (127.1.39.3:33435)
I20260926 19:31:55.357944 19408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:53727
I20260926 19:31:55.367050 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:31:55.384562 19733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.371470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43796 (local address 127.1.39.62:40767)
0926 19:31:55.371716 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.371727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.372190 (+ 463us) server_negotiation.cc:407] Connection header received
0926 19:31:55.372324 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.372328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.372390 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.372481 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:55.374072 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.374850 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.376218 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.376646 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.376753 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.379408 (+ 2655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:55.379413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:55.379416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:55.379446 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:55.382131 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.382879 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.382882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.382885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.383002 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.383350 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.383353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.383355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.383661 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:55.383865 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.384169 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.384299 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 19:31:55.388569 19408 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:31:55.395411 19408 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:31:55.428962 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.407312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.1.39.2:41653)
0926 19:31:55.418414 (+ 11102us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.418427 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.419680 (+ 1253us) server_negotiation.cc:407] Connection header received
0926 19:31:55.421665 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.421671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.421830 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.421975 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:55.422873 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.424387 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.426673 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.427809 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.427964 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.428186 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.428483 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.428654 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10948,"thread_start_us":10858,"threads_started":1}
I20260926 19:31:55.429039 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.406851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:41653 (local address 127.0.0.1:59436)
0926 19:31:55.419538 (+ 12687us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.419560 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.421375 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.422008 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.422015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.422058 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:55.422650 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.422665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.424515 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.424521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.426465 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.426481 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.428000 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.428006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.428234 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.428293 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.428442 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.428630 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12529,"mutex_wait_us":83,"thread_start_us":15319,"threads_started":1}
I20260926 19:31:55.429876 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.406783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53204)
0926 19:31:55.416842 (+ 10059us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.416896 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.417114 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.421042 (+ 3928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.421047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.421073 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:55.422285 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.422303 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.424335 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.424343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.426905 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.426925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.429032 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.429039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.429212 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.429245 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.429360 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.429493 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9733,"mutex_wait_us":113,"thread_start_us":12440,"threads_started":1}
I20260926 19:31:55.430039 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.406886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.1.39.1:46869)
0926 19:31:55.420279 (+ 13393us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.420290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.420322 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:31:55.420440 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.420452 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.420772 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.421011 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:55.422479 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.424155 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.427098 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.428560 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.428676 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.428888 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.429586 (+ 698us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.429750 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13180,"thread_start_us":13126,"threads_started":1}
I20260926 19:31:55.431532 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.407724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.1.39.3:33435)
0926 19:31:55.419003 (+ 11279us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.419014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.422876 (+ 3862us) server_negotiation.cc:407] Connection header received
0926 19:31:55.423168 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.423176 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.423387 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.423566 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:55.424228 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.425937 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.428559 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.430094 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.430221 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.430444 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.431012 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.431178 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11075,"thread_start_us":10893,"threads_started":1}
I20260926 19:31:55.432142 19878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.407228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:33435 (local address 127.0.0.1:50582)
0926 19:31:55.422756 (+ 15528us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.422774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.422975 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.423818 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.423821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.423838 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:55.424074 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.424080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.426141 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.426147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.428378 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.428393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.430628 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.430635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.430806 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.430842 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.430940 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.431078 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15400,"mutex_wait_us":22,"thread_start_us":10104,"threads_started":1}
I20260926 19:31:55.433709 19666 tablet_service.cc:1467] Processing CreateTablet for tablet 10234e9f31884f85b6b875b9c150932f (DEFAULT_TABLE table=test-table [id=9539ac4ea6db4ceabb235a6c3384b39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:55.433709 19532 tablet_service.cc:1467] Processing CreateTablet for tablet 10234e9f31884f85b6b875b9c150932f (DEFAULT_TABLE table=test-table [id=9539ac4ea6db4ceabb235a6c3384b39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:55.435155 19532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10234e9f31884f85b6b875b9c150932f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:55.435174 19666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10234e9f31884f85b6b875b9c150932f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:55.435134 19800 tablet_service.cc:1467] Processing CreateTablet for tablet 10234e9f31884f85b6b875b9c150932f (DEFAULT_TABLE table=test-table [id=9539ac4ea6db4ceabb235a6c3384b39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:55.436332 19800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10234e9f31884f85b6b875b9c150932f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:55.447081 19883 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Bootstrap starting.
I20260926 19:31:55.449643 19885 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Bootstrap starting.
I20260926 19:31:55.450475 19884 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Bootstrap starting.
I20260926 19:31:55.451629 19883 tablet_bootstrap.cc:654] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:55.452956 19883 log.cc:824] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:55.454486 19884 tablet_bootstrap.cc:654] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:55.455715 19884 log.cc:824] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:55.456353 19883 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: No bootstrap required, opened a new log
I20260926 19:31:55.456633 19883 ts_tablet_manager.cc:1397] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 19:31:55.456702 19885 tablet_bootstrap.cc:654] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:55.457749 19885 log.cc:824] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:55.458720 19884 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: No bootstrap required, opened a new log
I20260926 19:31:55.459033 19884 ts_tablet_manager.cc:1397] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 19:31:55.461259 19885 tablet_bootstrap.cc:492] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: No bootstrap required, opened a new log
I20260926 19:31:55.461526 19885 ts_tablet_manager.cc:1397] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260926 19:31:55.467967 19883 raft_consensus.cc:348] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.468504 19883 raft_consensus.cc:374] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:55.468636 19883 raft_consensus.cc:729] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6817fe4aafca47b4b2c55d777cea52ee, State: Initialized, Role: FOLLOWER
I20260926 19:31:55.468853 19885 raft_consensus.cc:348] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.469419 19883 consensus_queue.cc:260] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.469272 19885 raft_consensus.cc:374] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:55.469707 19885 raft_consensus.cc:729] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95c4d95deb34945955152789ca14993, State: Initialized, Role: FOLLOWER
I20260926 19:31:55.469942 19884 raft_consensus.cc:348] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.470567 19884 raft_consensus.cc:374] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:55.470439 19885 consensus_queue.cc:260] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.470738 19884 raft_consensus.cc:729] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9484f32eacbe41599419c7cda48dfa08, State: Initialized, Role: FOLLOWER
I20260926 19:31:55.472229 19885 ts_tablet_manager.cc:1428] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20260926 19:31:55.472973 19865 heartbeater.cc:503] Master 127.1.39.62:40767 was elected leader, sending a full tablet report...
I20260926 19:31:55.473091 19884 consensus_queue.cc:260] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.473672 19883 ts_tablet_manager.cc:1428] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260926 19:31:55.475713 19731 heartbeater.cc:503] Master 127.1.39.62:40767 was elected leader, sending a full tablet report...
I20260926 19:31:55.476375 19884 ts_tablet_manager.cc:1428] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260926 19:31:55.541565 19598 tablet.cc:2369] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:55.576579 19732 tablet.cc:2369] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:55.583709 19866 tablet.cc:2369] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:31:55.653120 19889 raft_consensus.cc:482] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:55.653491 19889 raft_consensus.cc:504] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.655400 19889 leader_election.cc:288] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95c4d95deb34945955152789ca14993 (127.1.39.1:46869), 9484f32eacbe41599419c7cda48dfa08 (127.1.39.2:41653)
I20260926 19:31:55.663404 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.656184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:58497 (local address 127.1.39.1:46869)
0926 19:31:55.656390 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.656413 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.656471 (+ 58us) server_negotiation.cc:407] Connection header received
0926 19:31:55.656555 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.656560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.656633 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.656724 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:55.657265 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.658394 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.660986 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.662276 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.662403 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.662506 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.663001 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.663144 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:31:55.663401 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.655997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:46327 (local address 127.1.39.2:41653)
0926 19:31:55.656235 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.656244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.656893 (+ 649us) server_negotiation.cc:407] Connection header received
0926 19:31:55.657093 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.657098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.657178 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.657264 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:55.657888 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.658572 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.660995 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.662260 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.662404 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.662514 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.663013 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.663144 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 19:31:55.663522 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.655693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.1.39.3:58497)
0926 19:31:55.655896 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.655917 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.656050 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.656799 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.656806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.656842 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:55.657120 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.657130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.658532 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.658539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.660814 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.660830 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.662441 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.662449 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.662729 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.662825 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.662926 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.663099 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":127,"spinlock_wait_cycles":22400}
I20260926 19:31:55.663633 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.655957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:41653 (local address 127.1.39.3:46327)
0926 19:31:55.656798 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.656813 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:55.656940 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:55.657288 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:55.657292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:55.657346 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:55.657670 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.657681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.658707 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.658712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.660815 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:55.660830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.662467 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.662472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:55.662735 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.662824 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 19:31:55.662926 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 19:31:55.663106 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":162,"thread_start_us":510,"threads_started":1}
I20260926 19:31:55.664629 19686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10234e9f31884f85b6b875b9c150932f" candidate_uuid: "6817fe4aafca47b4b2c55d777cea52ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9484f32eacbe41599419c7cda48dfa08" is_pre_election: true
I20260926 19:31:55.664671 19552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10234e9f31884f85b6b875b9c150932f" candidate_uuid: "6817fe4aafca47b4b2c55d777cea52ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95c4d95deb34945955152789ca14993" is_pre_election: true
I20260926 19:31:55.665161 19686 raft_consensus.cc:2454] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6817fe4aafca47b4b2c55d777cea52ee in term 0.
I20260926 19:31:55.665162 19552 raft_consensus.cc:2454] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6817fe4aafca47b4b2c55d777cea52ee in term 0.
I20260926 19:31:55.665877 19754 leader_election.cc:302] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6817fe4aafca47b4b2c55d777cea52ee, 9484f32eacbe41599419c7cda48dfa08, c95c4d95deb34945955152789ca14993; no voters:
I20260926 19:31:55.666270 19889 raft_consensus.cc:2781] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:31:55.666422 19889 raft_consensus.cc:482] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:31:55.666525 19889 raft_consensus.cc:3037] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:55.669005 19889 raft_consensus.cc:504] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.669739 19889 leader_election.cc:288] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [CANDIDATE]: Term 1 election: Requested vote from peers c95c4d95deb34945955152789ca14993 (127.1.39.1:46869), 9484f32eacbe41599419c7cda48dfa08 (127.1.39.2:41653)
I20260926 19:31:55.670185 19686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10234e9f31884f85b6b875b9c150932f" candidate_uuid: "6817fe4aafca47b4b2c55d777cea52ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9484f32eacbe41599419c7cda48dfa08"
I20260926 19:31:55.670221 19552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10234e9f31884f85b6b875b9c150932f" candidate_uuid: "6817fe4aafca47b4b2c55d777cea52ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95c4d95deb34945955152789ca14993"
I20260926 19:31:55.670426 19686 raft_consensus.cc:3037] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:55.670434 19552 raft_consensus.cc:3037] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:55.672588 19552 raft_consensus.cc:2454] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6817fe4aafca47b4b2c55d777cea52ee in term 1.
I20260926 19:31:55.672631 19686 raft_consensus.cc:2454] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6817fe4aafca47b4b2c55d777cea52ee in term 1.
I20260926 19:31:55.673004 19755 leader_election.cc:302] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6817fe4aafca47b4b2c55d777cea52ee, c95c4d95deb34945955152789ca14993; no voters:
I20260926 19:31:55.673368 19889 raft_consensus.cc:2781] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:55.673769 19889 raft_consensus.cc:686] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 1 LEADER]: Becoming Leader. State: Replica: 6817fe4aafca47b4b2c55d777cea52ee, State: Running, Role: LEADER
I20260926 19:31:55.674226 19889 consensus_queue.cc:237] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } }
I20260926 19:31:55.678666 19408 catalog_manager.cc:5495] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 6817fe4aafca47b4b2c55d777cea52ee (127.1.39.3). New cstate: current_term: 1 leader_uuid: "6817fe4aafca47b4b2c55d777cea52ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6817fe4aafca47b4b2c55d777cea52ee" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 33435 } health_report { overall_health: HEALTHY } } }
I20260926 19:31:55.727784 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.722295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50590 (local address 127.1.39.3:33435)
0926 19:31:55.722500 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.722504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.722522 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:31:55.722768 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.722772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.722847 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.722973 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:55.723323 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.724033 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.725501 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.725980 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.726133 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.726662 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:55.727209 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:31:55.727384 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.727516 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.727562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:31:55.736887 19686 raft_consensus.cc:1261] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 1 FOLLOWER]: Refusing update from remote peer 6817fe4aafca47b4b2c55d777cea52ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:55.736887 19552 raft_consensus.cc:1261] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 1 FOLLOWER]: Refusing update from remote peer 6817fe4aafca47b4b2c55d777cea52ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:55.738081 19893 consensus_queue.cc:1035] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95c4d95deb34945955152789ca14993" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:55.738499 19889 consensus_queue.cc:1035] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "9484f32eacbe41599419c7cda48dfa08" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 41653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:55.763077 19407 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:55.763383 19407 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:55.767390 19407 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 96b3ba254b6749078011b99f962f18d9: Sending DeleteTablet for 3 replicas of tablet 10234e9f31884f85b6b875b9c150932f
I20260926 19:31:55.768839 19666 tablet_service.cc:1514] Processing DeleteTablet for tablet 10234e9f31884f85b6b875b9c150932f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:55 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:59436
I20260926 19:31:55.768834 19532 tablet_service.cc:1514] Processing DeleteTablet for tablet 10234e9f31884f85b6b875b9c150932f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:55 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:53204
I20260926 19:31:55.769223 19800 tablet_service.cc:1514] Processing DeleteTablet for tablet 10234e9f31884f85b6b875b9c150932f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:55 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:50582
I20260926 19:31:55.774287 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19469
I20260926 19:31:55.782075 19906 tablet_replica.cc:330] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: stopping tablet replica
I20260926 19:31:55.782830 19906 raft_consensus.cc:2229] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:55.783766 19906 raft_consensus.cc:2258] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:55.786420 19906 ts_tablet_manager.cc:1905] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:55.795368 19906 ts_tablet_manager.cc:1918] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:55.795892 19906 log.cc:1196] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/10234e9f31884f85b6b875b9c150932f
I20260926 19:31:55.796886 19906 ts_tablet_manager.cc:1939] T 10234e9f31884f85b6b875b9c150932f P 9484f32eacbe41599419c7cda48dfa08: Deleting consensus metadata
I20260926 19:31:55.799372 19393 catalog_manager.cc:4841] TS 9484f32eacbe41599419c7cda48dfa08 (127.1.39.2:41653): tablet 10234e9f31884f85b6b875b9c150932f (table test-table [id=9539ac4ea6db4ceabb235a6c3384b39a]) successfully deleted
I20260926 19:31:55.802783 19908 tablet_replica.cc:330] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: stopping tablet replica
I20260926 19:31:55.803396 19908 raft_consensus.cc:2229] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:31:55.804255 19908 raft_consensus.cc:2258] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:55.806792 19908 ts_tablet_manager.cc:1905] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:31:55 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:31:55.812974 19907 tablet_replica.cc:330] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: stopping tablet replica
I20260926 19:31:55.813637 19907 raft_consensus.cc:2229] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:55.814321 19907 raft_consensus.cc:2258] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:55.816234 19908 ts_tablet_manager.cc:1918] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:55.816449 19908 log.cc:1196] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/10234e9f31884f85b6b875b9c150932f
I20260926 19:31:55.816682 19907 ts_tablet_manager.cc:1905] T 10234e9f31884f85b6b875b9c150932f P c95c4d95deb34945955152789ca14993: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:55.817036 19908 ts_tablet_manager.cc:1939] T 10234e9f31884f85b6b875b9c150932f P 6817fe4aafca47b4b2c55d777cea52ee: Deleting consensus metadata
I20260926 19:31:55.818877 19392 catalog_manager.cc:4841] TS 6817fe4aafca47b4b2c55d777cea52ee (127.1.39.3:33435): tablet 10234e9f31884f85b6b875b9c150932f (table test-table [id=9539ac4ea6db4ceabb235a6c3384b39a]) successfully deleted
I20260926 19:31:55.821251 19879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.777704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53208 (local address 127.1.39.1:46869)
0926 19:31:55.777924 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.777932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.801892 (+ 23960us) server_negotiation.cc:407] Connection header received
0926 19:31:55.802111 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.802125 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.802246 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.802363 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:55.804201 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.805394 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.807274 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.807922 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.808021 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.813610 (+ 5589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:55.813629 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:55.813650 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:55.813696 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:55.817897 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.818794 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.818802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.818805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.818967 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.819454 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.819460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.819464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.819938 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:55.820248 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.820595 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.820834 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:31:55.822022 19592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:55.927233 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19469
I20260926 19:31:55.946080 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19603
W20260926 19:31:55.946098 19394 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.39.1:46869 (error 108)
I20260926 19:31:55.949970 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.947673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53212)
0926 19:31:55.949696 (+ 2023us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:55.949804 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1900,"thread_start_us":1802,"threads_started":1}
W20260926 19:31:55.950335 19394 catalog_manager.cc:4584] TS c95c4d95deb34945955152789ca14993 (127.1.39.1:46869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10234e9f31884f85b6b875b9c150932f: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Sep 26 19:31:55 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:31:55.964051 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:55.947239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.1.39.2:41653)
0926 19:31:55.947466 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:55.947473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:55.947493 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:31:55.947563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:55.947568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:55.947665 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:55.947779 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:55.949513 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.950757 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.952138 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:55.952784 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:55.952913 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:55.957208 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:55.957226 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:55.957247 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:55.957322 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:55.960976 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.961724 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.961736 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.961740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.961939 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:55.962352 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:55.962357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:55.962389 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:55.962873 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:55.963159 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:55.963441 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 19:31:55.963671 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:31:55.964730 19726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:56.004217 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.003661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53214)
0926 19:31:56.003920 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.004029 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 19:31:56.005460 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.004998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53218)
0926 19:31:56.005168 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.005268 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 19:31:56.055137 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19603
I20260926 19:31:56.066097 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.065616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53230)
0926 19:31:56.065840 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.065938 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 19:31:56.067265 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.066880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53234)
0926 19:31:56.067037 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.067135 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 19:31:56.079308 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19737
Sep 26 19:31:56 dist-test-slave-9wb0 krb5kdc[19363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:31:56.099195 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.080157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.1.39.3:33435)
0926 19:31:56.080400 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:56.080409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:56.080479 (+ 70us) server_negotiation.cc:407] Connection header received
0926 19:31:56.080662 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:56.080666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:56.080737 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:56.080831 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:56.082644 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:56.083543 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:56.085474 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:56.085978 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:56.086095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:56.091594 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:56.091608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:56.091630 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:31:56.091680 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:56.096000 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:56.096751 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:56.096760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:56.096766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:56.096969 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:31:56.097435 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:31:56.097442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:31:56.097446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:31:56.097899 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:56.098218 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:56.098522 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 19:31:56.098757 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 19:31:56.099959 19860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:56.147979 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.147438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53248)
0926 19:31:56.147704 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.147813 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 19:31:56.149168 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.148765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53258)
0926 19:31:56.148953 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.149043 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 19:31:56.230105 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19737
I20260926 19:31:56.247154 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19376
I20260926 19:31:56.248330 19438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:56.362293 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.361652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:46869 (local address 127.0.0.1:53264)
0926 19:31:56.361880 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:56.361996 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.1:46869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 19:31:56.362746 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19376
2026-09-26T19:31:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4578 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T19:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:31:56Z Disabled control of system clock
I20260926 19:31:56.446936 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:34017
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:34017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:31:56.784698 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:56.785006 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:56.785089 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:56.793129 19919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:31:56.793241 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:56.793278 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:56.793402 19919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:31:56.793452 19919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:31:56.802955 19919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:34017
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:34017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:56.804247 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:56.806073 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:56.818225 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:56.818172 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:56.818255 19925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:56.819391 19919 server_base.cc:1034] running on GCE node
I20260926 19:31:56.820272 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:56.821830 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:56.823026 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790451116822982 us; error 43 us; skew 500 ppm
I20260926 19:31:56.823383 19919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:56.825770 19919 webserver.cc:460] Webserver started at http://127.1.39.62:36139/ using document root <none> and password file <none>
I20260926 19:31:56.826458 19919 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:56.826573 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:56.826911 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:56.829850 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "0ddcd678dfa74e698e3d5f2a24bff714"
format_stamp: "Formatted at 2026-09-26 19:31:56 on dist-test-slave-9wb0"
server_key: "56cd911c18cd49f23dbf275e0782d3ae"
server_key_iv: "6e95a373b056ef4d0474f4ebf422b4b7"
server_key_version: "encryptionkey@0"
I20260926 19:31:56.830691 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "0ddcd678dfa74e698e3d5f2a24bff714"
format_stamp: "Formatted at 2026-09-26 19:31:56 on dist-test-slave-9wb0"
server_key: "56cd911c18cd49f23dbf275e0782d3ae"
server_key_iv: "6e95a373b056ef4d0474f4ebf422b4b7"
server_key_version: "encryptionkey@0"
I20260926 19:31:56.835692 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:31:56.839147 19933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:56.840479 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:31:56.840723 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "0ddcd678dfa74e698e3d5f2a24bff714"
format_stamp: "Formatted at 2026-09-26 19:31:56 on dist-test-slave-9wb0"
server_key: "56cd911c18cd49f23dbf275e0782d3ae"
server_key_iv: "6e95a373b056ef4d0474f4ebf422b4b7"
server_key_version: "encryptionkey@0"
I20260926 19:31:56.840970 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:56.852806 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:56.853713 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:56.854004 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:56.872908 19919 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:34017
I20260926 19:31:56.872989 19984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:34017 every 8 connection(s)
I20260926 19:31:56.874325 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:31:56.877935 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:56.883925 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 19919
I20260926 19:31:56.884122 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:31:56.884644 1180 external_mini_cluster.cc:1428] Setting key 7ce7bb3632e763d817950d742da8f984
I20260926 19:31:56.889001 19985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: Bootstrap starting.
I20260926 19:31:56.892490 19985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:56.893762 19985 log.cc:824] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:56.897099 19985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: No bootstrap required, opened a new log
I20260926 19:31:56.898978 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:56.886675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.1.39.62:34017)
0926 19:31:56.887704 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:56.887715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:56.887756 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:31:56.889570 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:56.889593 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:56.890022 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:56.890345 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:56.890994 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:56.892243 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:56.894162 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:56.895282 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:56.895422 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:56.896631 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:56.896638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:56.896652 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:56.897210 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:56.897346 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:56.897378 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:56.897511 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:56.897732 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 19:31:56.898059 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":376,"threads_started":1}
I20260926 19:31:56.904685 19985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } }
I20260926 19:31:56.904994 19985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:56.905128 19985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddcd678dfa74e698e3d5f2a24bff714, State: Initialized, Role: FOLLOWER
I20260926 19:31:56.905675 19985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } }
I20260926 19:31:56.905900 19985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:31:56.906029 19985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:31:56.906236 19985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:56.908367 19985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } }
I20260926 19:31:56.908865 19985 leader_election.cc:302] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ddcd678dfa74e698e3d5f2a24bff714; no voters:
I20260926 19:31:56.909459 19985 leader_election.cc:288] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:31:56.909572 19990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:56.910360 19990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddcd678dfa74e698e3d5f2a24bff714, State: Running, Role: LEADER
I20260926 19:31:56.910835 19990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } }
I20260926 19:31:56.911480 19985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:31:56.913651 19991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } } }
I20260926 19:31:56.914152 19991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:56.914687 19992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ddcd678dfa74e698e3d5f2a24bff714. Latest consensus state: current_term: 1 leader_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddcd678dfa74e698e3d5f2a24bff714" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 34017 } } }
I20260926 19:31:56.914988 19992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714 [sys.catalog]: This master's current role is: LEADER
I20260926 19:31:56.916885 19999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:31:56.921837 19999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:31:56.929803 19999 catalog_manager.cc:1347] Generated new cluster ID: 10a308afc998433daa265fecc53a9727
I20260926 19:31:56.929925 19999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:31:56.943581 19999 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:31:56.944931 19999 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:31:56.954978 19999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: Generated new TSK 0
I20260926 19:31:56.956036 19999 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:31:56.974184 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--builtin_ntp_servers=127.1.39.20:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:31:57.309078 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:57.309453 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:57.309612 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:57.317370 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:57.317495 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:57.317677 20009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:31:57.326879 20009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:57.328198 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:57.330119 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:57.343132 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:57.343183 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:57.343343 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:57.344013 20009 server_base.cc:1034] running on GCE node
I20260926 19:31:57.344692 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:57.345752 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:57.346951 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790451117346894 us; error 55 us; skew 500 ppm
I20260926 19:31:57.347282 20009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:57.349608 20009 webserver.cc:460] Webserver started at http://127.1.39.1:45941/ using document root <none> and password file <none>
I20260926 19:31:57.350308 20009 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:57.350425 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:57.350754 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:57.353731 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "0fee7e9e8d904e059978c300125acb53"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "c2149f7a15d7108ebc05ee00be9fd83d"
server_key_iv: "b379972ee6ae038beb4166f7473bd5bb"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.354576 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "0fee7e9e8d904e059978c300125acb53"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "c2149f7a15d7108ebc05ee00be9fd83d"
server_key_iv: "b379972ee6ae038beb4166f7473bd5bb"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.359570 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:31:57.362910 20023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.364256 20009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:31:57.364439 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "0fee7e9e8d904e059978c300125acb53"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "c2149f7a15d7108ebc05ee00be9fd83d"
server_key_iv: "b379972ee6ae038beb4166f7473bd5bb"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.364651 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:57.382617 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:57.383441 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:57.383724 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:57.384691 20009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:57.386252 20009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:57.386338 20009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.386492 20009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:57.386595 20009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.420257 20009 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:40747
I20260926 19:31:57.420341 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:40747 every 8 connection(s)
I20260926 19:31:57.421643 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:31:57.430430 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20009
I20260926 19:31:57.430632 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:31:57.431171 1180 external_mini_cluster.cc:1428] Setting key e83eb5503ffd3aa4962fc42a94b5f217
I20260926 19:31:57.434222 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--builtin_ntp_servers=127.1.39.20:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:31:57.434779 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:57.424227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:58373 (local address 127.1.39.62:34017)
0926 19:31:57.424677 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:57.424682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:57.425806 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 19:31:57.427577 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:57.427583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:57.427682 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:57.427803 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:57.428805 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.429592 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.431856 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.432378 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.432509 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:57.433955 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:57.433959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:57.433963 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:57.433979 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:57.434022 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:57.434035 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:57.434115 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:57.434402 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 19:31:57.434506 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":242,"threads_started":1}
I20260926 19:31:57.436316 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:57.424737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:34017 (local address 127.1.39.1:58373)
0926 19:31:57.425655 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:57.425696 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:57.427320 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:57.428016 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:57.428024 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:57.428106 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:57.428640 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:57.428658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.429739 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.429751 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:57.431696 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:57.431711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.432607 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.432616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:57.432854 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:57.433617 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:57.433620 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:57.433671 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:57.433771 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:57.434142 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:57.434163 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:57.434172 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 19:31:57.434371 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 19:31:57.434784 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":273,"threads_started":1}
I20260926 19:31:57.437863 20136 heartbeater.cc:347] Connected to a master server at 127.1.39.62:34017
I20260926 19:31:57.438185 20136 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:57.439098 20136 heartbeater.cc:511] Master 127.1.39.62:34017 requested a full tablet report, sending...
I20260926 19:31:57.441355 19950 ts_manager.cc:194] Registered new tserver with Master: 0fee7e9e8d904e059978c300125acb53 (127.1.39.1:40747)
I20260926 19:31:57.442860 19950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.1:58373
W20260926 19:31:57.767261 20140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:57.767551 20140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:57.767701 20140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:57.775058 20140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:57.775200 20140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:57.775385 20140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:31:57.784358 20140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:57.785665 20140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:57.787389 20140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:57.800277 20145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:57.800246 20146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:57.800212 20148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:57.801116 20140 server_base.cc:1034] running on GCE node
I20260926 19:31:57.801776 20140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:57.802706 20140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:57.803913 20140 hybrid_clock.cc:648] HybridClock initialized: now 1790451117803870 us; error 33 us; skew 500 ppm
I20260926 19:31:57.804198 20140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:57.806388 20140 webserver.cc:460] Webserver started at http://127.1.39.2:43309/ using document root <none> and password file <none>
I20260926 19:31:57.807045 20140 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:57.807147 20140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:57.807451 20140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:57.810336 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "d5da6511d6fb45178e76d569cd855f13"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "fbd9babb0db0c6b11468dc5eb2ef54f8"
server_key_iv: "714996a1c47c8ff78feadf70885fd966"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.811118 20140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "d5da6511d6fb45178e76d569cd855f13"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "fbd9babb0db0c6b11468dc5eb2ef54f8"
server_key_iv: "714996a1c47c8ff78feadf70885fd966"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.816421 20140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:31:57.820127 20154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.821503 20140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:31:57.821669 20140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "d5da6511d6fb45178e76d569cd855f13"
format_stamp: "Formatted at 2026-09-26 19:31:57 on dist-test-slave-9wb0"
server_key: "fbd9babb0db0c6b11468dc5eb2ef54f8"
server_key_iv: "714996a1c47c8ff78feadf70885fd966"
server_key_version: "encryptionkey@0"
I20260926 19:31:57.821885 20140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:57.833156 20140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:57.833979 20140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:57.834256 20140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:57.835167 20140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:57.836593 20140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:57.836720 20140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.836843 20140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:57.836927 20140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:57.870841 20140 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:44111
I20260926 19:31:57.870893 20266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:44111 every 8 connection(s)
I20260926 19:31:57.872139 20140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:31:57.873224 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20140
I20260926 19:31:57.873405 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:31:57.873940 1180 external_mini_cluster.cc:1428] Setting key d1f39091279aec9b3e42f67498c57ed2
I20260926 19:31:57.877784 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--builtin_ntp_servers=127.1.39.20:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:31:57.893433 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:57.879337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:48785 (local address 127.1.39.62:34017)
0926 19:31:57.879513 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:57.879520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:57.882431 (+ 2911us) server_negotiation.cc:407] Connection header received
0926 19:31:57.885379 (+ 2948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:57.885388 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:57.885476 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:57.885594 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:57.887087 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.888214 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.890619 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.891049 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.891161 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:57.892589 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:57.892594 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:57.892598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:57.892618 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:57.892651 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:57.892664 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:57.892765 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:57.892990 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 19:31:57.893111 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:31:57.894493 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:57.878773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:34017 (local address 127.1.39.2:48785)
0926 19:31:57.882245 (+ 3472us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:57.882296 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:57.884975 (+ 2679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:57.885875 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:57.885889 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:57.886022 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:57.886871 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:57.886899 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.888385 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.888390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:57.890447 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:57.890465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:57.891254 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:57.891259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:57.891505 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:57.892243 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:57.892245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:57.892290 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:57.892372 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:57.892762 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:57.892773 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:57.892784 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 19:31:57.892990 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 19:31:57.893418 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2849,"thread_start_us":2622,"threads_started":1}
I20260926 19:31:57.896001 20267 heartbeater.cc:347] Connected to a master server at 127.1.39.62:34017
I20260926 19:31:57.896330 20267 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:57.897074 20267 heartbeater.cc:511] Master 127.1.39.62:34017 requested a full tablet report, sending...
I20260926 19:31:57.898697 19950 ts_manager.cc:194] Registered new tserver with Master: d5da6511d6fb45178e76d569cd855f13 (127.1.39.2:44111)
I20260926 19:31:57.899945 19950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.2:48785
W20260926 19:31:58.219599 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:31:58.219877 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:31:58.219966 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:31:58.227412 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:31:58.227523 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:31:58.227631 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:31:58.236873 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:34017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:31:58.238153 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:31:58.239919 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:31:58.253185 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:58.253150 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:31:58.253087 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:31:58.258934 20270 server_base.cc:1034] running on GCE node
I20260926 19:31:58.259598 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:31:58.260623 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:31:58.261859 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790451118261811 us; error 47 us; skew 500 ppm
I20260926 19:31:58.262197 20270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:31:58.264413 20270 webserver.cc:460] Webserver started at http://127.1.39.3:36655/ using document root <none> and password file <none>
I20260926 19:31:58.265107 20270 fs_manager.cc:359] Metadata directory not provided
I20260926 19:31:58.265236 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:31:58.265592 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:31:58.268565 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "7885fca61271491dbefb3de8eafe3094"
format_stamp: "Formatted at 2026-09-26 19:31:58 on dist-test-slave-9wb0"
server_key: "7ca70527701d0c2e3c8b4182e884c113"
server_key_iv: "575a13c502559da9af5f26ece3036829"
server_key_version: "encryptionkey@0"
I20260926 19:31:58.269495 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "7885fca61271491dbefb3de8eafe3094"
format_stamp: "Formatted at 2026-09-26 19:31:58 on dist-test-slave-9wb0"
server_key: "7ca70527701d0c2e3c8b4182e884c113"
server_key_iv: "575a13c502559da9af5f26ece3036829"
server_key_version: "encryptionkey@0"
I20260926 19:31:58.274748 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:31:58.278151 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:58.279438 20270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:31:58.279634 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "7885fca61271491dbefb3de8eafe3094"
format_stamp: "Formatted at 2026-09-26 19:31:58 on dist-test-slave-9wb0"
server_key: "7ca70527701d0c2e3c8b4182e884c113"
server_key_iv: "575a13c502559da9af5f26ece3036829"
server_key_version: "encryptionkey@0"
I20260926 19:31:58.279850 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:31:58.305250 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:31:58.306238 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:31:58.306557 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:31:58.307586 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:31:58.309118 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:31:58.309247 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:58.309397 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:31:58.309491 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:31:58.343703 20270 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:34551
I20260926 19:31:58.343791 20396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:34551 every 8 connection(s)
I20260926 19:31:58.345166 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:31:58.350433 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20270
I20260926 19:31:58.350622 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:31:58.351083 1180 external_mini_cluster.cc:1428] Setting key 568d2f0d5a37260416a16ba8c2aeeb39
I20260926 19:31:58.357721 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.347852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:45593 (local address 127.1.39.62:34017)
0926 19:31:58.348037 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.348041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.349383 (+ 1342us) server_negotiation.cc:407] Connection header received
0926 19:31:58.351342 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.351349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.351435 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.351594 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:58.352776 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.353763 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.355247 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.355660 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.355754 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.357017 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:58.357021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:58.357024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:58.357042 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:58.357073 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:58.357087 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:58.357164 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.357409 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.357483 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:31:58.358711 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.348336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:34017 (local address 127.1.39.3:45593)
0926 19:31:58.349198 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.349233 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.351049 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.351839 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.351847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.351938 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:31:58.352596 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.352622 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.353905 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.353909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.355125 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.355136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.355837 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.355841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.356029 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.356716 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:31:58.356719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:31:58.356756 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:31:58.356831 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:31:58.357174 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:31:58.357181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:31:58.357186 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.357386 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.357687 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":273,"threads_started":1}
I20260926 19:31:58.360029 20397 heartbeater.cc:347] Connected to a master server at 127.1.39.62:34017
I20260926 19:31:58.360322 20397 heartbeater.cc:464] Registering TS with master...
I20260926 19:31:58.361044 20397 heartbeater.cc:511] Master 127.1.39.62:34017 requested a full tablet report, sending...
I20260926 19:31:58.362555 19950 ts_manager.cc:194] Registered new tserver with Master: 7885fca61271491dbefb3de8eafe3094 (127.1.39.3:34551)
I20260926 19:31:58.363404 19950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.3:45593
I20260926 19:31:58.366381 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:31:58.376354 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.370653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.1.39.62:34017)
0926 19:31:58.370854 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.370858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.371308 (+ 450us) server_negotiation.cc:407] Connection header received
0926 19:31:58.371436 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.371463 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.371542 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.371636 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:58.372057 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.372777 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.374196 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.374620 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.374726 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.375723 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:58.375728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:58.375731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:58.375749 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:58.375783 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:58.375796 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:58.375921 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.376068 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.376112 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:31:58.380255 19950 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:31:58.387045 19950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:31:58.417574 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.400020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.1.39.1:40747)
0926 19:31:58.408073 (+ 8053us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.408082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.408121 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:31:58.408226 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.408234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.408525 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.408718 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:58.409588 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.411508 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.414756 (+ 3248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416172 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416305 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.416518 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.416951 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.417162 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7885,"thread_start_us":7759,"threads_started":1}
I20260926 19:31:58.417609 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.399452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:34551 (local address 127.0.0.1:39388)
0926 19:31:58.408076 (+ 8624us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.408100 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.408289 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.409062 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.409066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.409096 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:58.409658 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.409675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.411884 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.411890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.414228 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.414275 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416652 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.416897 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.416933 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.417002 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.417108 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8469,"mutex_wait_us":81,"thread_start_us":6418,"threads_started":1}
I20260926 19:31:58.417632 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.399414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40747 (local address 127.0.0.1:48426)
0926 19:31:58.406907 (+ 7493us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.406968 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.407194 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.408842 (+ 1648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.408853 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.408889 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:58.409420 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.409432 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.411639 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.411646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.414497 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.414511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416382 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.416618 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.416693 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.416839 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.417049 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6658,"mutex_wait_us":259,"thread_start_us":9164,"threads_started":1}
I20260926 19:31:58.417984 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.400539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.1.39.2:44111)
0926 19:31:58.407091 (+ 6552us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.407100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.408917 (+ 1817us) server_negotiation.cc:407] Connection header received
0926 19:31:58.409284 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.409294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.409559 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.409765 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:58.410324 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.411962 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.414756 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416175 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416285 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.416513 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.417402 (+ 889us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.417609 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6337,"thread_start_us":6212,"threads_started":1}
I20260926 19:31:58.418642 20410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.399924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:44111 (local address 127.0.0.1:56150)
0926 19:31:58.408814 (+ 8890us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.408833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.409017 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.409764 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.409768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.409793 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:58.410135 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.410147 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.412132 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.412138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.414504 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.414512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416796 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416841 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.417056 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.417093 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.417191 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.418216 (+ 1025us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8774,"mutex_wait_us":23,"thread_start_us":7822,"threads_started":1}
I20260926 19:31:58.418645 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.400160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.1.39.3:34551)
0926 19:31:58.407582 (+ 7422us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.407593 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.408191 (+ 598us) server_negotiation.cc:407] Connection header received
0926 19:31:58.408469 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.408477 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.408684 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.408832 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:58.409907 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.411686 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.414492 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.416085 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.416169 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.416451 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.418034 (+ 1583us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.418243 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7226,"thread_start_us":7239,"threads_started":1}
I20260926 19:31:58.422693 20071 tablet_service.cc:1467] Processing CreateTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 (DEFAULT_TABLE table=test-table [id=68a1d14aff034628b79486ea3abcafa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:58.422685 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 (DEFAULT_TABLE table=test-table [id=68a1d14aff034628b79486ea3abcafa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:58.422685 20202 tablet_service.cc:1467] Processing CreateTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 (DEFAULT_TABLE table=test-table [id=68a1d14aff034628b79486ea3abcafa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:31:58.424187 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138e53c5cb17478c8cf50632ebe49e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:58.424185 20202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138e53c5cb17478c8cf50632ebe49e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:58.424407 20071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 138e53c5cb17478c8cf50632ebe49e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:31:58.438599 20415 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Bootstrap starting.
I20260926 19:31:58.442258 20415 tablet_bootstrap.cc:654] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:58.443429 20417 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Bootstrap starting.
I20260926 19:31:58.443629 20415 log.cc:824] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:58.445176 20416 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Bootstrap starting.
I20260926 19:31:58.445817 20136 heartbeater.cc:503] Master 127.1.39.62:34017 was elected leader, sending a full tablet report...
I20260926 19:31:58.447745 20415 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: No bootstrap required, opened a new log
I20260926 19:31:58.447885 20417 tablet_bootstrap.cc:654] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:58.448083 20415 ts_tablet_manager.cc:1397] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260926 19:31:58.450116 20416 tablet_bootstrap.cc:654] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Neither blocks nor log segments found. Creating new log.
I20260926 19:31:58.450068 20417 log.cc:824] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:58.451675 20416 log.cc:824] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Log is configured to *not* fsync() on all Append() calls
I20260926 19:31:58.454814 20416 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: No bootstrap required, opened a new log
I20260926 19:31:58.455122 20416 ts_tablet_manager.cc:1397] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 19:31:58.455266 20417 tablet_bootstrap.cc:492] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: No bootstrap required, opened a new log
I20260926 19:31:58.455541 20417 ts_tablet_manager.cc:1397] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 19:31:58.459932 20415 raft_consensus.cc:348] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.460661 20415 raft_consensus.cc:374] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:58.460827 20415 raft_consensus.cc:729] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fee7e9e8d904e059978c300125acb53, State: Initialized, Role: FOLLOWER
I20260926 19:31:58.461614 20415 consensus_queue.cc:260] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.463652 20415 ts_tablet_manager.cc:1428] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260926 19:31:58.464771 20416 raft_consensus.cc:348] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.465338 20416 raft_consensus.cc:374] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:58.465476 20416 raft_consensus.cc:729] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7885fca61271491dbefb3de8eafe3094, State: Initialized, Role: FOLLOWER
I20260926 19:31:58.466195 20416 consensus_queue.cc:260] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.466658 20417 raft_consensus.cc:348] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.467237 20417 raft_consensus.cc:374] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:31:58.467389 20417 raft_consensus.cc:729] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5da6511d6fb45178e76d569cd855f13, State: Initialized, Role: FOLLOWER
I20260926 19:31:58.467520 20397 heartbeater.cc:503] Master 127.1.39.62:34017 was elected leader, sending a full tablet report...
I20260926 19:31:58.468139 20416 ts_tablet_manager.cc:1428] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Time spent starting tablet: real 0.013s user 0.000s sys 0.014s
I20260926 19:31:58.468303 20417 consensus_queue.cc:260] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.469543 20267 heartbeater.cc:503] Master 127.1.39.62:34017 was elected leader, sending a full tablet report...
I20260926 19:31:58.470254 20417 ts_tablet_manager.cc:1428] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260926 19:31:58.597034 20398 tablet.cc:2369] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:58.630386 20268 tablet.cc:2369] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:31:58.674237 20137 tablet.cc:2369] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:31:58.687230 20422 raft_consensus.cc:482] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:31:58.687546 20422 raft_consensus.cc:504] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.689273 20422 leader_election.cc:288] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0fee7e9e8d904e059978c300125acb53 (127.1.39.1:40747), d5da6511d6fb45178e76d569cd855f13 (127.1.39.2:44111)
I20260926 19:31:58.696167 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.689698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:34857 (local address 127.1.39.1:40747)
0926 19:31:58.689903 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.689909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.689933 (+ 24us) server_negotiation.cc:407] Connection header received
0926 19:31:58.690017 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.690025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.690151 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.690275 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:58.690804 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.691764 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.694062 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.695216 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.695348 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.695440 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.695751 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.695889 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:31:58.696274 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.689496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:40747 (local address 127.1.39.3:34857)
0926 19:31:58.689668 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.689696 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.689831 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.690350 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.690355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.690393 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:58.690625 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.690638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.691903 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.691906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.693881 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.693895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.695359 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.695363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.695558 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.695608 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.695702 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.695879 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260926 19:31:58.696856 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.690001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:53531 (local address 127.1.39.2:44111)
0926 19:31:58.690249 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.690256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.690660 (+ 404us) server_negotiation.cc:407] Connection header received
0926 19:31:58.690817 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.690822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.690914 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.691013 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:31:58.691497 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.692308 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.694505 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.695736 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.695853 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.695931 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.696437 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.696573 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 19:31:58.697510 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.689759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:44111 (local address 127.1.39.3:53531)
0926 19:31:58.690545 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0926 19:31:58.690564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:31:58.690683 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:31:58.691031 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:31:58.691034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:31:58.691053 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:31:58.691332 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.691338 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.692462 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.692468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.694328 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:31:58.694338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.696040 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.696046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:31:58.696231 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.696276 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 19:31:58.697041 (+ 765us) client_negotiation.cc:240] Negotiation successful
0926 19:31:58.697179 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":422,"threads_started":1}
I20260926 19:31:58.697510 20091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138e53c5cb17478c8cf50632ebe49e51" candidate_uuid: "7885fca61271491dbefb3de8eafe3094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fee7e9e8d904e059978c300125acb53" is_pre_election: true
I20260926 19:31:58.698082 20091 raft_consensus.cc:2454] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7885fca61271491dbefb3de8eafe3094 in term 0.
I20260926 19:31:58.698537 20222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138e53c5cb17478c8cf50632ebe49e51" candidate_uuid: "7885fca61271491dbefb3de8eafe3094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5da6511d6fb45178e76d569cd855f13" is_pre_election: true
I20260926 19:31:58.698833 20286 leader_election.cc:302] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fee7e9e8d904e059978c300125acb53, 7885fca61271491dbefb3de8eafe3094; no voters:
I20260926 19:31:58.698971 20222 raft_consensus.cc:2454] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7885fca61271491dbefb3de8eafe3094 in term 0.
I20260926 19:31:58.699240 20422 raft_consensus.cc:2781] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:31:58.699393 20422 raft_consensus.cc:482] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:31:58.699527 20422 raft_consensus.cc:3037] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:58.701828 20422 raft_consensus.cc:504] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.702461 20422 leader_election.cc:288] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [CANDIDATE]: Term 1 election: Requested vote from peers 0fee7e9e8d904e059978c300125acb53 (127.1.39.1:40747), d5da6511d6fb45178e76d569cd855f13 (127.1.39.2:44111)
I20260926 19:31:58.702924 20091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138e53c5cb17478c8cf50632ebe49e51" candidate_uuid: "7885fca61271491dbefb3de8eafe3094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fee7e9e8d904e059978c300125acb53"
I20260926 19:31:58.702984 20222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "138e53c5cb17478c8cf50632ebe49e51" candidate_uuid: "7885fca61271491dbefb3de8eafe3094" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5da6511d6fb45178e76d569cd855f13"
I20260926 19:31:58.703150 20091 raft_consensus.cc:3037] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:58.703222 20222 raft_consensus.cc:3037] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:31:58.705374 20091 raft_consensus.cc:2454] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7885fca61271491dbefb3de8eafe3094 in term 1.
I20260926 19:31:58.705485 20222 raft_consensus.cc:2454] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7885fca61271491dbefb3de8eafe3094 in term 1.
I20260926 19:31:58.705790 20286 leader_election.cc:302] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fee7e9e8d904e059978c300125acb53, 7885fca61271491dbefb3de8eafe3094; no voters:
I20260926 19:31:58.706053 20422 raft_consensus.cc:2781] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:31:58.706461 20422 raft_consensus.cc:686] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 1 LEADER]: Becoming Leader. State: Replica: 7885fca61271491dbefb3de8eafe3094, State: Running, Role: LEADER
I20260926 19:31:58.706933 20422 consensus_queue.cc:237] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } }
I20260926 19:31:58.711452 19949 catalog_manager.cc:5495] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7885fca61271491dbefb3de8eafe3094 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "7885fca61271491dbefb3de8eafe3094" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7885fca61271491dbefb3de8eafe3094" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 34551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 } health_report { overall_health: UNKNOWN } } }
I20260926 19:31:58.793648 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.788170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.1.39.3:34551)
0926 19:31:58.788337 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.788341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.788501 (+ 160us) server_negotiation.cc:407] Connection header received
0926 19:31:58.788623 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.788626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.788708 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.788852 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:31:58.789191 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.790097 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.791437 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.791843 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.791936 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.792410 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:31:58.792926 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:31:58.793059 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.793240 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.793276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 19:31:58.801447 20091 raft_consensus.cc:1261] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 1 FOLLOWER]: Refusing update from remote peer 7885fca61271491dbefb3de8eafe3094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:58.801517 20222 raft_consensus.cc:1261] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 1 FOLLOWER]: Refusing update from remote peer 7885fca61271491dbefb3de8eafe3094: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:31:58.802536 20425 consensus_queue.cc:1035] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fee7e9e8d904e059978c300125acb53" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:58.802948 20422 consensus_queue.cc:1035] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5da6511d6fb45178e76d569cd855f13" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 44111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:31:58.814944 20429 mvcc.cc:205] Tried to move back new op lower bound from 7333687782601039872 to 7333687782227226624. Current Snapshot: MvccSnapshot[applied={T|T < 7333687782601039872}]
I20260926 19:31:58.835741 19950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:58.836050 19950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:31:58.839558 19950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ddcd678dfa74e698e3d5f2a24bff714: Sending DeleteTablet for 3 replicas of tablet 138e53c5cb17478c8cf50632ebe49e51
I20260926 19:31:58.840698 20071 tablet_service.cc:1514] Processing DeleteTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:58 UTC) from {username='slave'} at 127.0.0.1:48426
I20260926 19:31:58.840739 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:58 UTC) from {username='slave'} at 127.0.0.1:39388
I20260926 19:31:58.841058 20202 tablet_service.cc:1514] Processing DeleteTablet for tablet 138e53c5cb17478c8cf50632ebe49e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:31:58 UTC) from {username='slave'} at 127.0.0.1:56150
I20260926 19:31:58.842983 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20009
I20260926 19:31:58.853154 20439 tablet_replica.cc:330] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: stopping tablet replica
I20260926 19:31:58.853860 20440 tablet_replica.cc:330] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: stopping tablet replica
I20260926 19:31:58.854450 20439 raft_consensus.cc:2229] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:58.854449 20440 raft_consensus.cc:2229] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:31:58.855114 20440 raft_consensus.cc:2258] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:58.855696 20439 raft_consensus.cc:2258] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:58.856017 20438 tablet_replica.cc:330] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: stopping tablet replica
I20260926 19:31:58.856463 20438 raft_consensus.cc:2229] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:31:58.857220 20438 raft_consensus.cc:2258] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:31:58.857854 20440 ts_tablet_manager.cc:1905] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:58.858397 20439 ts_tablet_manager.cc:1905] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:58.860350 20438 ts_tablet_manager.cc:1905] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:31:58.865832 20440 ts_tablet_manager.cc:1918] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:58.866073 20440 log.cc:1196] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/138e53c5cb17478c8cf50632ebe49e51
I20260926 19:31:58.866693 20440 ts_tablet_manager.cc:1939] T 138e53c5cb17478c8cf50632ebe49e51 P d5da6511d6fb45178e76d569cd855f13: Deleting consensus metadata
I20260926 19:31:58.867928 20439 ts_tablet_manager.cc:1918] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:58.868218 20439 log.cc:1196] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/138e53c5cb17478c8cf50632ebe49e51
I20260926 19:31:58.868860 20439 ts_tablet_manager.cc:1939] T 138e53c5cb17478c8cf50632ebe49e51 P 0fee7e9e8d904e059978c300125acb53: Deleting consensus metadata
I20260926 19:31:58.868893 19936 catalog_manager.cc:4841] TS d5da6511d6fb45178e76d569cd855f13 (127.1.39.2:44111): tablet 138e53c5cb17478c8cf50632ebe49e51 (table test-table [id=68a1d14aff034628b79486ea3abcafa1]) successfully deleted
I20260926 19:31:58.870754 19935 catalog_manager.cc:4841] TS 0fee7e9e8d904e059978c300125acb53 (127.1.39.1:40747): tablet 138e53c5cb17478c8cf50632ebe49e51 (table test-table [id=68a1d14aff034628b79486ea3abcafa1]) successfully deleted
I20260926 19:31:58.873409 20438 ts_tablet_manager.cc:1918] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:31:58.873484 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:58.844725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.1.39.1:40747)
0926 19:31:58.844902 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:58.844909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:58.860245 (+ 15336us) server_negotiation.cc:407] Connection header received
0926 19:31:58.860464 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:58.860469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:58.860573 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:58.860691 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:58.861171 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.862362 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.864317 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:58.865008 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:58.865133 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:58.871705 (+ 6572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:58.871715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:58.871732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:58.872499 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:58.872558 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:58.872583 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:58.872661 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:58.872944 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 19:31:58.873062 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:31:58.873842 20438 log.cc:1196] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/138e53c5cb17478c8cf50632ebe49e51
I20260926 19:31:58.874428 20438 ts_tablet_manager.cc:1939] T 138e53c5cb17478c8cf50632ebe49e51 P 7885fca61271491dbefb3de8eafe3094: Deleting consensus metadata
I20260926 19:31:58.874737 20131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:58.876509 19937 catalog_manager.cc:4841] TS 7885fca61271491dbefb3de8eafe3094 (127.1.39.3:34551): tablet 138e53c5cb17478c8cf50632ebe49e51 (table test-table [id=68a1d14aff034628b79486ea3abcafa1]) successfully deleted
I20260926 19:31:59.012212 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20009
I20260926 19:31:59.031271 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20140
I20260926 19:31:59.039016 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:59.032161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.1.39.2:44111)
0926 19:31:59.032365 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:59.032372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:59.032394 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:31:59.032463 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:59.032469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:59.032590 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:59.032701 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:59.033066 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:59.034218 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:59.035488 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:59.036153 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:59.036279 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:59.037365 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:59.037378 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:59.037398 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:59.038213 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:59.038274 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:59.038300 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:59.038417 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:59.038523 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 19:31:59.038641 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:31:59.039738 20262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:59.133029 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20140
I20260926 19:31:59.151417 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20270
I20260926 19:31:59.158687 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:31:59.152219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.1.39.3:34551)
0926 19:31:59.152436 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:31:59.152443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:31:59.152465 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:31:59.152697 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:31:59.152703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:31:59.152795 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:31:59.152919 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:31:59.153338 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:59.154151 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:59.155425 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:31:59.155866 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:31:59.156020 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:31:59.157164 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:31:59.157173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:31:59.157188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:31:59.157931 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:31:59.157979 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:31:59.157998 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:31:59.158140 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 19:31:59.158234 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 19:31:59.158334 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:31:59.159396 20392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:59.279892 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20270
I20260926 19:31:59.301239 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 19919
I20260926 19:31:59.302445 19980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:31:59.411500 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 19919
2026-09-26T19:31:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3046 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:31:59 dist-test-slave-9wb0 krb5kdc[20448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:31:59 dist-test-slave-9wb0 krb5kdc[20448](info): set up 2 sockets
Sep 26 19:31:59 dist-test-slave-9wb0 krb5kdc[20448](info): commencing operation
krb5kdc: starting...
W20260926 19:32:00.549611 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:00 dist-test-slave-9wb0 krb5kdc[20448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:00.825441 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38581
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:01.160296 20461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:01.160616 20461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:01.160777 20461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:01.168150 20461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:01.168267 20461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:01.168372 20461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:01.168453 20461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:01.168519 20461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:01.178674 20461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:01.180039 20461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:01.181494 20461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:01.193369 20466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:01.193481 20467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:01.193261 20469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:01.194408 20461 server_base.cc:1034] running on GCE node
I20260926 19:32:01.195218 20461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:01.196666 20461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:01.197897 20461 hybrid_clock.cc:648] HybridClock initialized: now 1790451121197858 us; error 52 us; skew 500 ppm
I20260926 19:32:01.198256 20461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:01 dist-test-slave-9wb0 krb5kdc[20448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:01.208107 20461 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:01.209832 20461 webserver.cc:460] Webserver started at http://127.1.39.62:44603/ using document root <none> and password file <none>
I20260926 19:32:01.210559 20461 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:01.210704 20461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:01.211052 20461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:01.214110 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "6989bb4324e96b095fc9d8e02d3f905b"
server_key_iv: "16b763710276657a9c0ef942ce02461c"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.214854 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "6989bb4324e96b095fc9d8e02d3f905b"
server_key_iv: "16b763710276657a9c0ef942ce02461c"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.220077 20461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:01.223778 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:01.225185 20461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260926 19:32:01.225394 20461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "6989bb4324e96b095fc9d8e02d3f905b"
server_key_iv: "16b763710276657a9c0ef942ce02461c"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.225605 20461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:01.260640 20461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:01.264914 20461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:01.265230 20461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:01.284282 20461 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:38581
I20260926 19:32:01.284319 20527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:38581 every 8 connection(s)
I20260926 19:32:01.285759 20461 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:01.289455 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:01.291929 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20461
I20260926 19:32:01.292106 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:01.292619 1180 external_mini_cluster.cc:1428] Setting key 43a391690ec3412375e3f2ca0715ba71
I20260926 19:32:01.302184 20528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: Bootstrap starting.
I20260926 19:32:01.306093 20528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:01.307456 20528 log.cc:824] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:32:01 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:01.310618 20528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: No bootstrap required, opened a new log
I20260926 19:32:01.316821 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:01.294570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.1.39.62:38581)
0926 19:32:01.295408 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:01.295419 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:01.295464 (+ 45us) server_negotiation.cc:407] Connection header received
0926 19:32:01.297144 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:01.297165 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:01.297605 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:01.297899 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:01.299743 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.301072 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.302941 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.303680 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.303819 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:01.309375 (+ 5556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:01.309386 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:01.309411 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:01.309448 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:01.312638 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:01.313391 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:01.313411 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:01.313418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:01.313592 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:01.314011 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:01.314014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:01.314017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:01.314570 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:01.314938 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:01.315369 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 19:32:01.315801 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":359,"threads_started":1}
I20260926 19:32:01.318602 20528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } }
I20260926 19:32:01.318992 20528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:01.319095 20528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efc7cd1e6b14ff0a497f0ab1e7e62c3, State: Initialized, Role: FOLLOWER
I20260926 19:32:01.319746 20528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } }
I20260926 19:32:01.320016 20528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:01.320147 20528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:01.320310 20528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:01.322556 20528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } }
I20260926 19:32:01.323035 20528 leader_election.cc:302] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9efc7cd1e6b14ff0a497f0ab1e7e62c3; no voters:
I20260926 19:32:01.323841 20528 leader_election.cc:288] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:01.323938 20533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:01.324710 20533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [term 1 LEADER]: Becoming Leader. State: Replica: 9efc7cd1e6b14ff0a497f0ab1e7e62c3, State: Running, Role: LEADER
I20260926 19:32:01.325253 20533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } }
I20260926 19:32:01.325958 20528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:01.328032 20534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } } }
I20260926 19:32:01.328408 20534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:01.328917 20535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9efc7cd1e6b14ff0a497f0ab1e7e62c3. Latest consensus state: current_term: 1 leader_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9efc7cd1e6b14ff0a497f0ab1e7e62c3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38581 } } }
I20260926 19:32:01.329227 20535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:01.329586 20541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:01.336575 20541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:01.346221 20541 catalog_manager.cc:1347] Generated new cluster ID: 0d1b6e0d9aac42d1961d05b0b5967431
I20260926 19:32:01.346370 20541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:01.360422 20541 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:01.361835 20541 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:01.368079 20541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: Generated new TSK 0
I20260926 19:32:01.369119 20541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.39.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
Entry for principal kudu/127.1.39.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab.
I20260926 19:32:01.428493 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:01.761615 20554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:01.761895 20554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:01.762059 20554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:01.769346 20554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:01.769452 20554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:01.769630 20554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:32:01.779333 20554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.1.keytab
--principal=kudu/127.1.39.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:01.780652 20554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:01.781963 20554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:01.794116 20559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:01.794804 20560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:01.795652 20562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:01.796211 20554 server_base.cc:1034] running on GCE node
I20260926 19:32:01.796854 20554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:01.797861 20554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:01.799062 20554 hybrid_clock.cc:648] HybridClock initialized: now 1790451121799017 us; error 35 us; skew 500 ppm
I20260926 19:32:01.799368 20554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:01 dist-test-slave-9wb0 krb5kdc[20448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:01.802832 20554 init.cc:377] Logged in from keytab as kudu/127.1.39.1@KRBTEST.COM (short username kudu)
I20260926 19:32:01.804349 20554 webserver.cc:460] Webserver started at http://127.1.39.1:34983/ using document root <none> and password file <none>
I20260926 19:32:01.805060 20554 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:01.805205 20554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:01.805557 20554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:01.808422 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "a2779bf1d80047edaaa0883a3ed6e0c6"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "c700e861420d70cd69a5befa6ca25225"
server_key_iv: "a44494dcdaaf8d6e7a7a68e9162e49a5"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.809213 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "a2779bf1d80047edaaa0883a3ed6e0c6"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "c700e861420d70cd69a5befa6ca25225"
server_key_iv: "a44494dcdaaf8d6e7a7a68e9162e49a5"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.814814 20554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:32:01.818396 20569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:01.819936 20554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:32:01.820148 20554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "a2779bf1d80047edaaa0883a3ed6e0c6"
format_stamp: "Formatted at 2026-09-26 19:32:01 on dist-test-slave-9wb0"
server_key: "c700e861420d70cd69a5befa6ca25225"
server_key_iv: "a44494dcdaaf8d6e7a7a68e9162e49a5"
server_key_version: "encryptionkey@0"
I20260926 19:32:01.820367 20554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:01.834662 20554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:01.839058 20554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:01.839360 20554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:01.840361 20554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:01.841822 20554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:01.841936 20554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:01.842103 20554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:01.842187 20554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:01.877836 20554 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:33213
I20260926 19:32:01.877875 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:33213 every 8 connection(s)
I20260926 19:32:01.879240 20554 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:32:01.885370 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20554
I20260926 19:32:01.885600 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:32:01.886147 1180 external_mini_cluster.cc:1428] Setting key ed2ac24b68275ae7438f94d04688780f
Sep 26 19:32:01 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.1@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:01.899704 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:01.881935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:50851 (local address 127.1.39.62:38581)
0926 19:32:01.882401 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:01.882406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:01.883398 (+ 992us) server_negotiation.cc:407] Connection header received
0926 19:32:01.885402 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:01.885406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:01.885474 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:01.885616 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:01.887530 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.888408 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.890040 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.890456 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.890596 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:01.894043 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:01.894048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:01.894052 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:01.894082 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:01.896671 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:01.897435 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:01.897448 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:01.897450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:01.897571 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:01.898011 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:01.898020 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:01.898022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:01.898281 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:01.898438 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:01.899122 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 19:32:01.899354 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20260926 19:32:01.900725 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:01.882476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38581 (local address 127.1.39.1:50851)
0926 19:32:01.883222 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:01.883263 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:01.885042 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:01.885867 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:01.885879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:01.886616 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:01.887317 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:01.887342 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.888553 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.888564 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:01.889900 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:01.889912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:01.890686 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:01.890694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:01.890931 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:01.891608 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:01.891631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:01.893826 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:01.896828 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:01.896837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:01.896856 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:01.897281 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:01.897689 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:01.897695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:01.897698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:01.897890 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:01.898408 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:01.898415 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:01.898758 (+ 343us) client_negotiation.cc:769] Sending connection context
0926 19:32:01.899014 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 19:32:01.899540 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":304,"threads_started":1}
I20260926 19:32:01.902359 20682 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38581
I20260926 19:32:01.902700 20682 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:01.903610 20682 heartbeater.cc:511] Master 127.1.39.62:38581 requested a full tablet report, sending...
I20260926 19:32:01.906087 20493 ts_manager.cc:194] Registered new tserver with Master: a2779bf1d80047edaaa0883a3ed6e0c6 (127.1.39.1:33213)
I20260926 19:32:01.908233 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.1@KRBTEST.COM'} at 127.1.39.1:50851
WARNING: no policy specified for kudu/127.1.39.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
Entry for principal kudu/127.1.39.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab.
I20260926 19:32:01.949772 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:02.277559 20688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:02.277837 20688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:02.278002 20688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:02.285348 20688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:02.285460 20688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:02.285687 20688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:32:02.295756 20688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.2.keytab
--principal=kudu/127.1.39.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:02.296989 20688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:02.298316 20688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:02.311916 20693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:02.311877 20696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:02.312671 20694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:02.312700 20688 server_base.cc:1034] running on GCE node
I20260926 19:32:02.313534 20688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:02.314455 20688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:02.315685 20688 hybrid_clock.cc:648] HybridClock initialized: now 1790451122315629 us; error 42 us; skew 500 ppm
I20260926 19:32:02.315997 20688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:02 dist-test-slave-9wb0 krb5kdc[20448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:02.319065 20688 init.cc:377] Logged in from keytab as kudu/127.1.39.2@KRBTEST.COM (short username kudu)
I20260926 19:32:02.320549 20688 webserver.cc:460] Webserver started at http://127.1.39.2:38363/ using document root <none> and password file <none>
I20260926 19:32:02.321251 20688 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:02.321426 20688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:02.321744 20688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:02.324586 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "3897276d1c8442199f48a5436f4122d6"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "392da392de0e900b2c74f3cefa0eb15d"
server_key_iv: "73f288c0cceb128e4bb134b1fb985590"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.325474 20688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "3897276d1c8442199f48a5436f4122d6"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "392da392de0e900b2c74f3cefa0eb15d"
server_key_iv: "73f288c0cceb128e4bb134b1fb985590"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.330695 20688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 19:32:02.334084 20703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.335490 20688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:02.335673 20688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "3897276d1c8442199f48a5436f4122d6"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "392da392de0e900b2c74f3cefa0eb15d"
server_key_iv: "73f288c0cceb128e4bb134b1fb985590"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.335861 20688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:02.382701 20688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:02.386822 20688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:02.387219 20688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:02.388211 20688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:02.389619 20688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:02.389719 20688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.389883 20688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:02.389961 20688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.423033 20688 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:40601
I20260926 19:32:02.423085 20815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:40601 every 8 connection(s)
I20260926 19:32:02.424341 20688 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:32:02.426097 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20688
I20260926 19:32:02.426283 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:32:02.426775 1180 external_mini_cluster.cc:1428] Setting key 130789b8f424ba21065ed9e4d0249b77
Sep 26 19:32:02 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.2@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:02.446795 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:02.427265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:38365 (local address 127.1.39.62:38581)
0926 19:32:02.427762 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:02.427769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:02.428827 (+ 1058us) server_negotiation.cc:407] Connection header received
0926 19:32:02.430958 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:02.430964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:02.431054 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:02.431172 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:02.432669 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.433467 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.435364 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.435805 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.435938 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:02.440195 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:02.440201 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:02.440221 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:02.440256 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:02.443175 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:02.444198 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:02.444202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:02.444205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:02.444346 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:02.444890 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:02.444894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:02.444896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:02.445185 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:02.445383 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:02.446315 (+ 932us) server_negotiation.cc:299] Negotiation successful
0926 19:32:02.446511 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":407,"threads_started":1}
I20260926 19:32:02.448418 20819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:02.427861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38581 (local address 127.1.39.2:38365)
0926 19:32:02.428663 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:02.428713 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:02.430697 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:02.431412 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:02.431419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:02.431912 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:02.432471 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:02.432488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.433613 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.433619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:02.435227 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:02.435239 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.436025 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.436034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:02.436292 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:02.436991 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:02.437013 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:02.439930 (+ 2917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:02.443330 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:02.443342 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:02.443361 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:02.444066 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:02.444482 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:02.444489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:02.444496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:02.444753 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:02.445363 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:02.445376 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:02.445910 (+ 534us) client_negotiation.cc:769] Sending connection context
0926 19:32:02.446241 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 19:32:02.446868 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":371,"threads_started":1}
I20260926 19:32:02.450021 20816 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38581
I20260926 19:32:02.450348 20816 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:02.451085 20816 heartbeater.cc:511] Master 127.1.39.62:38581 requested a full tablet report, sending...
I20260926 19:32:02.452651 20493 ts_manager.cc:194] Registered new tserver with Master: 3897276d1c8442199f48a5436f4122d6 (127.1.39.2:40601)
I20260926 19:32:02.453612 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.2@KRBTEST.COM'} at 127.1.39.2:38365
WARNING: no policy specified for kudu/127.1.39.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
Entry for principal kudu/127.1.39.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab.
I20260926 19:32:02.491078 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:02.818182 20822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:02.818512 20822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:02.818694 20822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:02.826234 20822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:02.826468 20822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:02.826773 20822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:32:02.837211 20822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.3.keytab
--principal=kudu/127.1.39.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:02.838624 20822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:02.839980 20822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:02.853209 20828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:02.853142 20827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:02.853246 20830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:02.853928 20822 server_base.cc:1034] running on GCE node
I20260926 19:32:02.854481 20822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:02.855407 20822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:02.856606 20822 hybrid_clock.cc:648] HybridClock initialized: now 1790451122856560 us; error 45 us; skew 500 ppm
I20260926 19:32:02.856873 20822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:02 dist-test-slave-9wb0 krb5kdc[20448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:02.860126 20822 init.cc:377] Logged in from keytab as kudu/127.1.39.3@KRBTEST.COM (short username kudu)
I20260926 19:32:02.861784 20822 webserver.cc:460] Webserver started at http://127.1.39.3:37103/ using document root <none> and password file <none>
I20260926 19:32:02.862519 20822 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:02.862689 20822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:02.863027 20822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:02.866245 20822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "717257451a2847daa4b466109c4c0653"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "eb24afe9e517e9bb36c6923ba74878dc"
server_key_iv: "d0202744152323dd30baba4727bbfca2"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.867116 20822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "717257451a2847daa4b466109c4c0653"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "eb24afe9e517e9bb36c6923ba74878dc"
server_key_iv: "d0202744152323dd30baba4727bbfca2"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.872736 20822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:02.876536 20837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.878185 20822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:02.878376 20822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "717257451a2847daa4b466109c4c0653"
format_stamp: "Formatted at 2026-09-26 19:32:02 on dist-test-slave-9wb0"
server_key: "eb24afe9e517e9bb36c6923ba74878dc"
server_key_iv: "d0202744152323dd30baba4727bbfca2"
server_key_version: "encryptionkey@0"
I20260926 19:32:02.878666 20822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:02.912117 20682 heartbeater.cc:503] Master 127.1.39.62:38581 was elected leader, sending a full tablet report...
I20260926 19:32:02.926306 20822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:02.931241 20822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:02.931556 20822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:02.932523 20822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:02.933939 20822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:02.934043 20822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.934196 20822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:02.934273 20822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:02.970947 20822 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:38069
I20260926 19:32:02.970983 20949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:38069 every 8 connection(s)
I20260926 19:32:02.972304 20822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:32:02.977994 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 20822
I20260926 19:32:02.978183 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:32:02.978689 1180 external_mini_cluster.cc:1428] Setting key c10e85c3cf3dc3911cecb8118d6252f6
Sep 26 19:32:02 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.3@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:02.993374 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:02.975007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:41509 (local address 127.1.39.62:38581)
0926 19:32:02.975456 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:02.975460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:02.976566 (+ 1106us) server_negotiation.cc:407] Connection header received
0926 19:32:02.978970 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:02.978977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:02.979098 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:02.979242 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:02.981241 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.982261 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.983815 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.984238 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.984340 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:02.987780 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:02.987785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:02.987788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:02.987819 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:02.990307 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:02.990989 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:02.990993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:02.990995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:02.991115 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:02.991590 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:02.991593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:02.991595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:02.991845 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:02.991961 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:02.992765 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 19:32:02.992968 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":216,"threads_started":1}
I20260926 19:32:02.994499 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:02.975566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38581 (local address 127.1.39.3:41509)
0926 19:32:02.976325 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:02.976375 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:02.978611 (+ 2236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:02.979519 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:02.979551 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:02.980251 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:02.981048 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:02.981081 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.982409 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.982413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:02.983679 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:02.983697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:02.984447 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:02.984452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:02.984652 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:02.985428 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:02.985465 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:02.987596 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:02.990430 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:02.990436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:02.990449 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:02.990885 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:02.991210 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:02.991214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:02.991216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:02.991473 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:02.991958 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:02.991963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:02.992340 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 19:32:02.992643 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 19:32:02.993185 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260926 19:32:02.995726 20950 heartbeater.cc:347] Connected to a master server at 127.1.39.62:38581
I20260926 19:32:02.996039 20950 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:02.996752 20950 heartbeater.cc:511] Master 127.1.39.62:38581 requested a full tablet report, sending...
I20260926 19:32:02.998131 20493 ts_manager.cc:194] Registered new tserver with Master: 717257451a2847daa4b466109c4c0653 (127.1.39.3:38069)
I20260926 19:32:02.998895 20493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.39.3@KRBTEST.COM'} at 127.1.39.3:41509
I20260926 19:32:03.006747 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:32:03.023594 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.010944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.1.39.62:38581)
0926 19:32:03.011194 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.011200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.011512 (+ 312us) server_negotiation.cc:407] Connection header received
0926 19:32:03.011657 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.011660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.011734 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.011815 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:03.013424 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.014278 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.015642 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.016076 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.016190 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.018602 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:03.018607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:03.018610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:03.018642 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:03.021241 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.021896 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.021900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.021902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.022013 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.022324 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.022327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.022329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.022634 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:03.022817 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.023079 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.023246 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 19:32:03.027493 20493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:32:03.034677 20493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:32:03.060508 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.046970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:40601 (local address 127.0.0.1:42950)
0926 19:32:03.047656 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:03.047707 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:03.047889 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:03.049203 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:03.049210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:03.049247 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:03.049744 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.049761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.052673 (+ 2912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.052682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.056184 (+ 3502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.056209 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.058989 (+ 2780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.058999 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.059310 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.059404 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 19:32:03.059549 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 19:32:03.059829 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":10,"thread_start_us":295,"threads_started":1}
I20260926 19:32:03.061089 20965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.048327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33213 (local address 127.0.0.1:44990)
0926 19:32:03.049118 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:03.049136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:03.049236 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:03.050547 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:03.050554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:03.050588 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:03.051110 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.051128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.054262 (+ 3134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.054271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.057435 (+ 3164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.057458 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.060130 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.060138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.060426 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.060491 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 19:32:03.060574 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 19:32:03.060736 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":490,"threads_started":1}
I20260926 19:32:03.062637 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.048943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.1.39.1:33213)
0926 19:32:03.049870 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.049879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.049907 (+ 28us) server_negotiation.cc:407] Connection header received
0926 19:32:03.050003 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.050011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.050297 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.051392 (+ 1095us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:03.051746 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.054058 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.057745 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.059816 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.061418 (+ 1602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.061782 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.061878 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.062171 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":598,"threads_started":1}
I20260926 19:32:03.062731 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.047125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.1.39.2:40601)
0926 19:32:03.048492 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.048502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.048540 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:32:03.048652 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.048659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.048953 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.049188 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:03.049927 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.052470 (+ 2543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.056513 (+ 4043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.058690 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.061635 (+ 2945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.061913 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.062012 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.062262 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1195,"thread_start_us":1078,"threads_started":1}
I20260926 19:32:03.067071 20617 tablet_service.cc:1467] Processing CreateTablet for tablet d3a075e55bcd469abac51f7013a3ca19 (DEFAULT_TABLE table=test-table [id=42120244176e4c3d90cb31b6c79ffb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:03.067092 20751 tablet_service.cc:1467] Processing CreateTablet for tablet d3a075e55bcd469abac51f7013a3ca19 (DEFAULT_TABLE table=test-table [id=42120244176e4c3d90cb31b6c79ffb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:03.068337 20617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a075e55bcd469abac51f7013a3ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:03.068395 20751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a075e55bcd469abac51f7013a3ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:03.071897 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.050355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:38069 (local address 127.0.0.1:42176)
0926 19:32:03.061241 (+ 10886us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:03.061262 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:03.061417 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:03.064368 (+ 2951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:03.064373 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:03.064408 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:03.064596 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.064611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.066624 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.066629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.069342 (+ 2713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.069357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.071113 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.071119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.071296 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.071330 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 19:32:03.071431 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:32:03.071579 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10752,"thread_start_us":13859,"threads_started":1}
I20260926 19:32:03.072197 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.050772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.1.39.3:38069)
0926 19:32:03.063395 (+ 12623us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.063407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.063443 (+ 36us) server_negotiation.cc:407] Connection header received
0926 19:32:03.063542 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.063550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.063857 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.064060 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:03.064822 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.066445 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.069499 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.070937 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.071071 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.071296 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.071481 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.071651 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12439,"thread_start_us":12275,"threads_started":1}
I20260926 19:32:03.080194 20969 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Bootstrap starting.
I20260926 19:32:03.082087 20970 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Bootstrap starting.
I20260926 19:32:03.081839 20885 tablet_service.cc:1467] Processing CreateTablet for tablet d3a075e55bcd469abac51f7013a3ca19 (DEFAULT_TABLE table=test-table [id=42120244176e4c3d90cb31b6c79ffb92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:03.083123 20885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3a075e55bcd469abac51f7013a3ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:03.084367 20969 tablet_bootstrap.cc:654] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:03.085542 20970 tablet_bootstrap.cc:654] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:03.085745 20969 log.cc:824] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:03.086826 20970 log.cc:824] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:03.089241 20969 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: No bootstrap required, opened a new log
I20260926 19:32:03.089540 20969 ts_tablet_manager.cc:1397] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260926 19:32:03.089809 20970 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: No bootstrap required, opened a new log
I20260926 19:32:03.090063 20970 ts_tablet_manager.cc:1397] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260926 19:32:03.096923 20973 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Bootstrap starting.
I20260926 19:32:03.096885 20969 raft_consensus.cc:348] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.097388 20969 raft_consensus.cc:374] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:03.097488 20969 raft_consensus.cc:729] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3897276d1c8442199f48a5436f4122d6, State: Initialized, Role: FOLLOWER
I20260926 19:32:03.098099 20969 consensus_queue.cc:260] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.099431 20816 heartbeater.cc:503] Master 127.1.39.62:38581 was elected leader, sending a full tablet report...
I20260926 19:32:03.100201 20969 ts_tablet_manager.cc:1428] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260926 19:32:03.100365 20970 raft_consensus.cc:348] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.101051 20970 raft_consensus.cc:374] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:03.101636 20970 raft_consensus.cc:729] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2779bf1d80047edaaa0883a3ed6e0c6, State: Initialized, Role: FOLLOWER
I20260926 19:32:03.102223 20970 consensus_queue.cc:260] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.103855 20970 ts_tablet_manager.cc:1428] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260926 19:32:03.104965 20973 tablet_bootstrap.cc:654] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:03.106339 20973 log.cc:824] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:03.111756 20973 tablet_bootstrap.cc:492] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: No bootstrap required, opened a new log
I20260926 19:32:03.111982 20973 ts_tablet_manager.cc:1397] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260926 19:32:03.118713 20973 raft_consensus.cc:348] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.119038 20973 raft_consensus.cc:374] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:03.119105 20973 raft_consensus.cc:729] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717257451a2847daa4b466109c4c0653, State: Initialized, Role: FOLLOWER
I20260926 19:32:03.119580 20973 consensus_queue.cc:260] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.120501 20950 heartbeater.cc:503] Master 127.1.39.62:38581 was elected leader, sending a full tablet report...
I20260926 19:32:03.120954 20973 ts_tablet_manager.cc:1428] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
W20260926 19:32:03.131734 20683 tablet.cc:2369] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 19:32:03.178391 20817 tablet.cc:2369] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:03.208266 20977 raft_consensus.cc:482] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:32:03.208600 20977 raft_consensus.cc:504] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.210403 20977 leader_election.cc:288] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3897276d1c8442199f48a5436f4122d6 (127.1.39.2:40601), a2779bf1d80047edaaa0883a3ed6e0c6 (127.1.39.1:33213)
I20260926 19:32:03.216799 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.210995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:41533 (local address 127.1.39.2:40601)
0926 19:32:03.211218 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.211227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.211248 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:32:03.211346 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.211360 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.211450 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.211548 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:03.211915 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.212719 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.214737 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.215948 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.216076 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.216170 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.216466 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.216575 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 19:32:03.216914 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.210645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:40601 (local address 127.1.39.3:41533)
0926 19:32:03.210872 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:03.210896 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:03.211037 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:03.211583 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:03.211587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:03.211613 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:03.211766 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.211773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.212827 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.212832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.214579 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.214591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.216113 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.216119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.216305 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.216340 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:32:03.216441 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 19:32:03.216611 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":1}
I20260926 19:32:03.218142 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.210886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:34099 (local address 127.1.39.1:33213)
0926 19:32:03.211130 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.211140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.211798 (+ 658us) server_negotiation.cc:407] Connection header received
0926 19:32:03.211994 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.211998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.212056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.212151 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:03.212787 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.213905 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.216103 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.217332 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.217473 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.217562 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.217878 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.217978 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:32:03.218007 20771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a075e55bcd469abac51f7013a3ca19" candidate_uuid: "717257451a2847daa4b466109c4c0653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3897276d1c8442199f48a5436f4122d6" is_pre_election: true
I20260926 19:32:03.218329 20978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.210811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:33213 (local address 127.1.39.3:34099)
0926 19:32:03.211678 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:03.211709 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:03.211845 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:03.212163 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:03.212167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:03.212189 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:03.212611 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.212627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.214069 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.214073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.215913 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:03.215928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.217539 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.217545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:03.217718 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.217755 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 19:32:03.217853 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 19:32:03.217973 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":478,"threads_started":1}
I20260926 19:32:03.218578 20771 raft_consensus.cc:2454] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717257451a2847daa4b466109c4c0653 in term 0.
I20260926 19:32:03.219272 20839 leader_election.cc:302] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3897276d1c8442199f48a5436f4122d6, 717257451a2847daa4b466109c4c0653; no voters:
I20260926 19:32:03.219470 20637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a075e55bcd469abac51f7013a3ca19" candidate_uuid: "717257451a2847daa4b466109c4c0653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" is_pre_election: true
I20260926 19:32:03.219732 20977 raft_consensus.cc:2781] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:32:03.219924 20977 raft_consensus.cc:482] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:32:03.220031 20977 raft_consensus.cc:3037] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:03.220029 20637 raft_consensus.cc:2454] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 717257451a2847daa4b466109c4c0653 in term 0.
I20260926 19:32:03.222460 20977 raft_consensus.cc:504] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.223061 20977 leader_election.cc:288] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [CANDIDATE]: Term 1 election: Requested vote from peers 3897276d1c8442199f48a5436f4122d6 (127.1.39.2:40601), a2779bf1d80047edaaa0883a3ed6e0c6 (127.1.39.1:33213)
I20260926 19:32:03.223507 20637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a075e55bcd469abac51f7013a3ca19" candidate_uuid: "717257451a2847daa4b466109c4c0653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6"
I20260926 19:32:03.223476 20771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a075e55bcd469abac51f7013a3ca19" candidate_uuid: "717257451a2847daa4b466109c4c0653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3897276d1c8442199f48a5436f4122d6"
I20260926 19:32:03.223709 20771 raft_consensus.cc:3037] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:03.223709 20637 raft_consensus.cc:3037] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 0 FOLLOWER]: Advancing to term 1
W20260926 19:32:03.225620 20951 tablet.cc:2369] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:03.226994 20771 raft_consensus.cc:2454] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717257451a2847daa4b466109c4c0653 in term 1.
I20260926 19:32:03.226994 20637 raft_consensus.cc:2454] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 717257451a2847daa4b466109c4c0653 in term 1.
I20260926 19:32:03.227427 20838 leader_election.cc:302] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 717257451a2847daa4b466109c4c0653, a2779bf1d80047edaaa0883a3ed6e0c6; no voters:
I20260926 19:32:03.227771 20977 raft_consensus.cc:2781] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:03.228228 20977 raft_consensus.cc:686] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 1 LEADER]: Becoming Leader. State: Replica: 717257451a2847daa4b466109c4c0653, State: Running, Role: LEADER
I20260926 19:32:03.228753 20977 consensus_queue.cc:237] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } }
I20260926 19:32:03.233186 20492 catalog_manager.cc:5495] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 717257451a2847daa4b466109c4c0653 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "717257451a2847daa4b466109c4c0653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "717257451a2847daa4b466109c4c0653" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 38069 } health_report { overall_health: HEALTHY } } }
I20260926 19:32:03.264007 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.258408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42188 (local address 127.1.39.3:38069)
0926 19:32:03.258627 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.258634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.258732 (+ 98us) server_negotiation.cc:407] Connection header received
0926 19:32:03.258980 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.258984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.259063 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.259217 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:03.259638 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.260336 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.261886 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.262294 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.262425 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.262941 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:03.263445 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:03.263609 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.263662 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.263743 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:32:03.272293 20637 raft_consensus.cc:1261] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 1 FOLLOWER]: Refusing update from remote peer 717257451a2847daa4b466109c4c0653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:03.272296 20771 raft_consensus.cc:1261] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 1 FOLLOWER]: Refusing update from remote peer 717257451a2847daa4b466109c4c0653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:03.273379 20977 consensus_queue.cc:1035] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3897276d1c8442199f48a5436f4122d6" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 40601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:03.273694 20979 consensus_queue.cc:1035] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2779bf1d80047edaaa0883a3ed6e0c6" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:03.287997 20982 mvcc.cc:205] Tried to move back new op lower bound from 7333687800912691200 to 7333687800749232128. Current Snapshot: MvccSnapshot[applied={T|T < 7333687800912691200}]
I20260926 19:32:03.294505 20983 mvcc.cc:205] Tried to move back new op lower bound from 7333687800912691200 to 7333687800749232128. Current Snapshot: MvccSnapshot[applied={T|T < 7333687800912691200}]
I20260926 19:32:03.307257 20492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:32:03.307564 20492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:32:03.311408 20492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9efc7cd1e6b14ff0a497f0ab1e7e62c3: Sending DeleteTablet for 3 replicas of tablet d3a075e55bcd469abac51f7013a3ca19
I20260926 19:32:03.312639 20751 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a075e55bcd469abac51f7013a3ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:42950
I20260926 19:32:03.312667 20617 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a075e55bcd469abac51f7013a3ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:44990
I20260926 19:32:03.312960 20885 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a075e55bcd469abac51f7013a3ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:03 UTC) from {username='kudu', principal='kudu/127.1.39.62@KRBTEST.COM'} at 127.0.0.1:42176
I20260926 19:32:03.315189 20992 tablet_replica.cc:330] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: stopping tablet replica
I20260926 19:32:03.315734 20992 raft_consensus.cc:2229] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:32:03.316311 20992 raft_consensus.cc:2258] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:03.316711 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20554
I20260926 19:32:03.319766 20995 tablet_replica.cc:330] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: stopping tablet replica
I20260926 19:32:03.320222 20995 raft_consensus.cc:2229] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:32:03.320998 20995 raft_consensus.cc:2258] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:03.321051 20992 ts_tablet_manager.cc:1905] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:32:03.327409 20993 tablet_replica.cc:330] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: stopping tablet replica
I20260926 19:32:03.327877 20993 raft_consensus.cc:2229] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:32:03.327977 20995 ts_tablet_manager.cc:1905] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:32:03.328444 20993 raft_consensus.cc:2258] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:03.329668 20992 ts_tablet_manager.cc:1918] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:03.329864 20992 log.cc:1196] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/d3a075e55bcd469abac51f7013a3ca19
I20260926 19:32:03.330503 20992 ts_tablet_manager.cc:1939] T d3a075e55bcd469abac51f7013a3ca19 P 3897276d1c8442199f48a5436f4122d6: Deleting consensus metadata
I20260926 19:32:03.332585 20478 catalog_manager.cc:4841] TS 3897276d1c8442199f48a5436f4122d6 (127.1.39.2:40601): tablet d3a075e55bcd469abac51f7013a3ca19 (table test-table [id=42120244176e4c3d90cb31b6c79ffb92]) successfully deleted
I20260926 19:32:03.335335 20995 ts_tablet_manager.cc:1918] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:03.335500 20995 log.cc:1196] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/d3a075e55bcd469abac51f7013a3ca19
I20260926 19:32:03.336053 20995 ts_tablet_manager.cc:1939] T d3a075e55bcd469abac51f7013a3ca19 P 717257451a2847daa4b466109c4c0653: Deleting consensus metadata
I20260926 19:32:03.336338 20993 ts_tablet_manager.cc:1905] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 19:32:03 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.1@KRBTEST.COM
I20260926 19:32:03.338208 20479 catalog_manager.cc:4841] TS 717257451a2847daa4b466109c4c0653 (127.1.39.3:38069): tablet d3a075e55bcd469abac51f7013a3ca19 (table test-table [id=42120244176e4c3d90cb31b6c79ffb92]) successfully deleted
I20260926 19:32:03.342984 20993 ts_tablet_manager.cc:1918] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:03.343201 20993 log.cc:1196] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/d3a075e55bcd469abac51f7013a3ca19
I20260926 19:32:03.343853 20993 ts_tablet_manager.cc:1939] T d3a075e55bcd469abac51f7013a3ca19 P a2779bf1d80047edaaa0883a3ed6e0c6: Deleting consensus metadata
I20260926 19:32:03.345810 20477 catalog_manager.cc:4841] TS a2779bf1d80047edaaa0883a3ed6e0c6 (127.1.39.1:33213): tablet d3a075e55bcd469abac51f7013a3ca19 (table test-table [id=42120244176e4c3d90cb31b6c79ffb92]) successfully deleted
I20260926 19:32:03.347004 20966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.317859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44994 (local address 127.1.39.1:33213)
0926 19:32:03.318132 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.318141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.318398 (+ 257us) server_negotiation.cc:407] Connection header received
0926 19:32:03.318690 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.318697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.318805 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.318891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:03.321481 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.323358 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.328985 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.329588 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.329673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.339345 (+ 9672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:03.339356 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:03.339370 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:03.339407 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:03.343329 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.344316 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.344325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.344329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.344521 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.344999 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.345004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.345008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.345510 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:03.345817 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.346399 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.346620 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 19:32:03.347723 20677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:03.478964 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20554
I20260926 19:32:03.498299 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20688
Sep 26 19:32:03 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.2@KRBTEST.COM
I20260926 19:32:03.515988 20964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.499153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.1.39.2:40601)
0926 19:32:03.499370 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.499377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.499400 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:32:03.499523 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.499530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.499632 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.499744 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:03.500971 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.502147 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.503494 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.504114 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.504242 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.508998 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:03.509017 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:03.509037 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:03.509092 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:03.512859 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.513802 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.513811 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.513815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.513965 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.514362 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.514368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.514371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.514868 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:03.515170 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.515400 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.515616 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:32:03.516870 20811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:03.610759 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20688
I20260926 19:32:03.634711 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20822
Sep 26 19:32:03 dist-test-slave-9wb0 krb5kdc[20448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.3@KRBTEST.COM
I20260926 19:32:03.652645 20968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:03.635571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.1.39.3:38069)
0926 19:32:03.635738 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:03.635742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:03.635760 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:32:03.635941 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:03.635945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:03.636009 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:03.636101 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:03.637957 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.638734 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.640620 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:03.641249 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:03.641449 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:03.646554 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:03.646567 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:03.646584 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:03.646619 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:03.649816 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.650637 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.650645 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.650647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.650797 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:03.651199 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:03.651203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:03.651205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:03.651541 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:03.651792 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:03.652111 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 19:32:03.652308 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:32:03.653396 20945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:03.777560 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20822
I20260926 19:32:03.797793 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 20461
I20260926 19:32:03.798899 20523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:03.913834 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 20461
2026-09-26T19:32:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4504 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T19:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:03Z Disabled control of system clock
I20260926 19:32:03.997715 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37119
--webserver_interface=127.1.39.62
--webserver_port=0
--builtin_ntp_servers=127.1.39.20:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:32:04.333582 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:04.333925 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:04.334084 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:04.341591 21003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:04.341725 21003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:04.341795 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:04.341897 21003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:04.341964 21003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:04.351217 21003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:37119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:04.352573 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:04.354451 21003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:04.366312 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:04.366396 21008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:04.366356 21009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:04.367265 21003 server_base.cc:1034] running on GCE node
I20260926 19:32:04.368019 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:04.369457 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:04.370693 21003 hybrid_clock.cc:648] HybridClock initialized: now 1790451124370644 us; error 52 us; skew 500 ppm
I20260926 19:32:04.371034 21003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:32:04.373278 21003 webserver.cc:460] Webserver started at http://127.1.39.62:46563/ using document root <none> and password file <none>
I20260926 19:32:04.374030 21003 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:04.374166 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:04.374488 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:04.377406 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "ff2d4479a75e437495f56034f512eb56"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "898801fb9d7c77a3da57b4d95283eba9"
server_key_iv: "82aac793fea586e2cf5c828282e4bcd2"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.378259 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "ff2d4479a75e437495f56034f512eb56"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "898801fb9d7c77a3da57b4d95283eba9"
server_key_iv: "82aac793fea586e2cf5c828282e4bcd2"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.383478 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:32:04.387029 21017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:04.388356 21003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 19:32:04.388552 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "ff2d4479a75e437495f56034f512eb56"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "898801fb9d7c77a3da57b4d95283eba9"
server_key_iv: "82aac793fea586e2cf5c828282e4bcd2"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.388784 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:04.407337 21003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:04.408281 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:04.408599 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:04.427838 21003 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:37119
I20260926 19:32:04.427851 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:37119 every 8 connection(s)
I20260926 19:32:04.429225 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:04.431342 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21003
I20260926 19:32:04.431520 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:04.432076 1180 external_mini_cluster.cc:1428] Setting key a3a22bd1b7565d89f07d9ef378a9c183
I20260926 19:32:04.433845 21069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:04.447237 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:04.434159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.1.39.62:37119)
0926 19:32:04.435184 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:04.435199 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:04.435245 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:32:04.437130 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:04.437155 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:04.437781 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:04.438196 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:04.438725 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:04.440742 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:04.442201 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:04.443198 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:04.443388 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:04.444467 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:04.444480 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:04.444502 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:04.445227 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:04.445347 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:04.445377 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:04.445501 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:04.445682 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:32:04.446019 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":333,"threads_started":1}
I20260926 19:32:04.448427 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: Bootstrap starting.
I20260926 19:32:04.451431 21069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:04.452383 21069 log.cc:824] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:04.454988 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: No bootstrap required, opened a new log
I20260926 19:32:04.461975 21069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } }
I20260926 19:32:04.462342 21069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:04.462457 21069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2d4479a75e437495f56034f512eb56, State: Initialized, Role: FOLLOWER
I20260926 19:32:04.463086 21069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } }
I20260926 19:32:04.463315 21069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:04.463441 21069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:04.463611 21069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:04.465816 21069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } }
I20260926 19:32:04.466315 21069 leader_election.cc:302] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff2d4479a75e437495f56034f512eb56; no voters:
I20260926 19:32:04.466876 21069 leader_election.cc:288] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:04.467041 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:04.467821 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [term 1 LEADER]: Becoming Leader. State: Replica: ff2d4479a75e437495f56034f512eb56, State: Running, Role: LEADER
I20260926 19:32:04.468359 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } }
I20260926 19:32:04.468981 21069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:04.471485 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff2d4479a75e437495f56034f512eb56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } } }
I20260926 19:32:04.471951 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:04.474097 21083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:04.473932 21076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff2d4479a75e437495f56034f512eb56. Latest consensus state: current_term: 1 leader_uuid: "ff2d4479a75e437495f56034f512eb56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff2d4479a75e437495f56034f512eb56" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37119 } } }
I20260926 19:32:04.474416 21076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:04.479317 21083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:04.490522 21083 catalog_manager.cc:1347] Generated new cluster ID: 8092c8bc0039417c8587fd6f091caa58
I20260926 19:32:04.490654 21083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:04.517218 21083 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:04.519090 21083 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:04.531399 21083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: Generated new TSK 0
I20260926 19:32:04.532464 21083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:04.544770 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.1:0
--local_ip_for_outbound_sockets=127.1.39.1
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--builtin_ntp_servers=127.1.39.20:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 19:32:04.881573 21093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:04.881866 21093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:04.882011 21093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:04.889400 21093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:04.889513 21093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:04.889719 21093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.1
I20260926 19:32:04.899250 21093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.39.1
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.1
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:04.900601 21093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:04.902483 21093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:04.915551 21098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:04.915613 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:04.915616 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:04.916625 21093 server_base.cc:1034] running on GCE node
I20260926 19:32:04.917204 21093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:04.918192 21093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:04.919399 21093 hybrid_clock.cc:648] HybridClock initialized: now 1790451124919359 us; error 26 us; skew 500 ppm
I20260926 19:32:04.919724 21093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:32:04.921968 21093 webserver.cc:460] Webserver started at http://127.1.39.1:44569/ using document root <none> and password file <none>
I20260926 19:32:04.922706 21093 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:04.922850 21093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:04.923204 21093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:04.926156 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data/instance:
uuid: "bfe40c4608ec4718a34300090a6eac0b"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "8aa04aaf6891cdc52273e467379e11af"
server_key_iv: "1f297b949e8d7b4851351f1a1f825cc3"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.926949 21093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance:
uuid: "bfe40c4608ec4718a34300090a6eac0b"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "8aa04aaf6891cdc52273e467379e11af"
server_key_iv: "1f297b949e8d7b4851351f1a1f825cc3"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.932102 21093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:04.935969 21107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:04.937354 21093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:32:04.937525 21093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal
uuid: "bfe40c4608ec4718a34300090a6eac0b"
format_stamp: "Formatted at 2026-09-26 19:32:04 on dist-test-slave-9wb0"
server_key: "8aa04aaf6891cdc52273e467379e11af"
server_key_iv: "1f297b949e8d7b4851351f1a1f825cc3"
server_key_version: "encryptionkey@0"
I20260926 19:32:04.937758 21093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:04.979846 21093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:04.980743 21093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:04.981022 21093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:04.982067 21093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:04.983620 21093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:04.983747 21093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:04.983882 21093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:04.983970 21093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.017570 21093 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.1:34173
I20260926 19:32:05.017664 21219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.1:34173 every 8 connection(s)
I20260926 19:32:05.018951 21093 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/data/info.pb
I20260926 19:32:05.024819 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21093
I20260926 19:32:05.024991 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal/instance
I20260926 19:32:05.025563 1180 external_mini_cluster.cc:1428] Setting key a08a608542bbe7ef0859ce4d1db43b85
I20260926 19:32:05.028697 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.2:0
--local_ip_for_outbound_sockets=127.1.39.2
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--builtin_ntp_servers=127.1.39.20:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:32:05.034135 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.021611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.1:36287 (local address 127.1.39.62:37119)
0926 19:32:05.022121 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:05.022129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:05.023404 (+ 1275us) server_negotiation.cc:407] Connection header received
0926 19:32:05.025660 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:05.025666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:05.025763 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:05.025861 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:05.027175 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.028017 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.030635 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.031203 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.031303 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.033000 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:05.033005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:05.033008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:05.033028 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:05.033071 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:05.033084 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:05.033158 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:05.033482 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 19:32:05.033565 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260926 19:32:05.035351 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.022214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37119 (local address 127.1.39.1:36287)
0926 19:32:05.023194 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:05.023230 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:05.025259 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:05.026151 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:05.026165 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:05.026287 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:05.026985 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.027010 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.028183 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.028188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.030464 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.030483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.031452 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.031461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.031801 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.032605 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:32:05.032611 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:05.032674 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:32:05.032791 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:05.033183 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:05.033197 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:05.033207 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 19:32:05.033426 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 19:32:05.033903 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":314,"threads_started":1}
I20260926 19:32:05.036996 21220 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37119
I20260926 19:32:05.037384 21220 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:05.038349 21220 heartbeater.cc:511] Master 127.1.39.62:37119 requested a full tablet report, sending...
I20260926 19:32:05.040992 21034 ts_manager.cc:194] Registered new tserver with Master: bfe40c4608ec4718a34300090a6eac0b (127.1.39.1:34173)
I20260926 19:32:05.043251 21034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.1:36287
W20260926 19:32:05.361977 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:05.362275 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:05.362362 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:05.369539 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:05.369649 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:05.369751 21224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.2
I20260926 19:32:05.378877 21224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.39.2
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.2
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:05.380105 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:05.381892 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:05.395349 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:05.395406 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:05.395437 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:05.396137 21224 server_base.cc:1034] running on GCE node
I20260926 19:32:05.396718 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:05.397761 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:05.399016 21224 hybrid_clock.cc:648] HybridClock initialized: now 1790451125398952 us; error 53 us; skew 500 ppm
I20260926 19:32:05.399341 21224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:32:05.401671 21224 webserver.cc:460] Webserver started at http://127.1.39.2:37429/ using document root <none> and password file <none>
I20260926 19:32:05.402377 21224 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:05.402508 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:05.402834 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:05.405831 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data/instance:
uuid: "da3c36c094e845e9a6809fa76aacc6ae"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "02b60222426aff3e26822fc45e7f9af9"
server_key_iv: "e32fb175e025694075328652264ac684"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.407050 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance:
uuid: "da3c36c094e845e9a6809fa76aacc6ae"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "02b60222426aff3e26822fc45e7f9af9"
server_key_iv: "e32fb175e025694075328652264ac684"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.412325 21224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:05.416023 21238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.417520 21224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:05.417707 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal
uuid: "da3c36c094e845e9a6809fa76aacc6ae"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "02b60222426aff3e26822fc45e7f9af9"
server_key_iv: "e32fb175e025694075328652264ac684"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.417932 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:05.435815 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:05.436667 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:05.436957 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:05.438052 21224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:05.439584 21224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:05.439702 21224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.439831 21224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:05.439921 21224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.477759 21224 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.2:46231
I20260926 19:32:05.477835 21350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.2:46231 every 8 connection(s)
I20260926 19:32:05.479108 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/data/info.pb
I20260926 19:32:05.479578 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21224
I20260926 19:32:05.479779 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal/instance
I20260926 19:32:05.480399 1180 external_mini_cluster.cc:1428] Setting key 289c28086840d5140ca805ee7455b0d3
I20260926 19:32:05.483361 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.39.3:0
--local_ip_for_outbound_sockets=127.1.39.3
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--builtin_ntp_servers=127.1.39.20:32919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 19:32:05.497378 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.482166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.2:57757 (local address 127.1.39.62:37119)
0926 19:32:05.482376 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:05.482382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:05.484515 (+ 2133us) server_negotiation.cc:407] Connection header received
0926 19:32:05.487096 (+ 2581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:05.487106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:05.487205 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:05.487294 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:05.489243 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.490394 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.493398 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.493999 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.494111 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.495997 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:05.496006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:05.496013 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:05.496039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:05.496117 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:05.496183 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:05.496316 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:05.496707 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 19:32:05.496825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:32:05.500649 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.482679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37119 (local address 127.1.39.2:57757)
0926 19:32:05.484377 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:05.484414 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:05.486752 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:05.487854 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:05.487867 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:05.487991 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:05.488846 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.488877 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.490766 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.490774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.493126 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.493145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.494228 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.494237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.494588 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.495603 (+ 1015us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:32:05.495609 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:05.495663 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:32:05.495776 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:05.496405 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:05.496418 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:05.496428 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 19:32:05.498077 (+ 1649us) client_negotiation.cc:240] Negotiation successful
0926 19:32:05.498622 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":339,"threads_started":1}
I20260926 19:32:05.502683 21351 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37119
I20260926 19:32:05.503029 21351 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:05.504004 21351 heartbeater.cc:511] Master 127.1.39.62:37119 requested a full tablet report, sending...
I20260926 19:32:05.505664 21034 ts_manager.cc:194] Registered new tserver with Master: da3c36c094e845e9a6809fa76aacc6ae (127.1.39.2:46231)
I20260926 19:32:05.506690 21034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.2:57757
W20260926 19:32:05.833331 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:05.833819 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:05.834005 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:05.845038 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:05.845189 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:05.845408 21354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.39.3
I20260926 19:32:05.854902 21354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:32919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.39.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.39.3
--webserver_port=0
--tserver_master_addrs=127.1.39.62:37119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.39.3
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:05.856210 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:05.858018 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:05.874560 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:05.875238 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:05.875259 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:05.875702 21354 server_base.cc:1034] running on GCE node
I20260926 19:32:05.876320 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:05.877529 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:05.878791 21354 hybrid_clock.cc:648] HybridClock initialized: now 1790451125878726 us; error 62 us; skew 500 ppm
I20260926 19:32:05.879144 21354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 19:32:05.882040 21354 webserver.cc:460] Webserver started at http://127.1.39.3:38267/ using document root <none> and password file <none>
I20260926 19:32:05.882761 21354 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:05.882877 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:05.883221 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:05.886523 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data/instance:
uuid: "a74759bed29143d9939c48d33c7881c2"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "b36bb0fc26a6cb97055376a3929d9d8c"
server_key_iv: "0bdf24b32321f4cf2adfc42c39125b39"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.887373 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance:
uuid: "a74759bed29143d9939c48d33c7881c2"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "b36bb0fc26a6cb97055376a3929d9d8c"
server_key_iv: "0bdf24b32321f4cf2adfc42c39125b39"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.892823 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:05.896621 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.898151 21354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:05.898322 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal
uuid: "a74759bed29143d9939c48d33c7881c2"
format_stamp: "Formatted at 2026-09-26 19:32:05 on dist-test-slave-9wb0"
server_key: "b36bb0fc26a6cb97055376a3929d9d8c"
server_key_iv: "0bdf24b32321f4cf2adfc42c39125b39"
server_key_version: "encryptionkey@0"
I20260926 19:32:05.898555 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:05.910774 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:05.911689 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:05.911942 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:05.913013 21354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 19:32:05.914662 21354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 19:32:05.914748 21354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.914911 21354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 19:32:05.914995 21354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:05.967938 21354 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.3:36761
I20260926 19:32:05.968055 21480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.3:36761 every 8 connection(s)
I20260926 19:32:05.969571 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/data/info.pb
I20260926 19:32:05.972252 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21354
I20260926 19:32:05.972467 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal/instance
I20260926 19:32:05.972945 1180 external_mini_cluster.cc:1428] Setting key 99419ad60c8ce1bd2f795c89b8b7b7a6
I20260926 19:32:05.985131 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.974151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:40757 (local address 127.1.39.62:37119)
0926 19:32:05.974337 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:05.974344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:05.974932 (+ 588us) server_negotiation.cc:407] Connection header received
0926 19:32:05.977264 (+ 2332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:05.977273 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:05.977406 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:05.977540 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:05.978599 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.979948 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.981638 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.982531 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.982705 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.984074 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:05.984084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:05.984092 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:05.984122 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:05.984172 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:05.984194 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:05.984309 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:05.984506 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 19:32:05.984640 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 19:32:05.985870 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:05.973488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:37119 (local address 127.1.39.3:40757)
0926 19:32:05.974773 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:05.974824 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:05.976982 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:05.977748 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:05.977757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:05.977844 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:05.978420 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.978438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.980127 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.980131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.981460 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:05.981473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:05.982750 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:05.982755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:05.982950 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:05.983734 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 19:32:05.983736 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:05.983781 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 19:32:05.983857 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:05.984298 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:05.984305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:05.984311 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 19:32:05.984456 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 19:32:05.984752 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":643,"threads_started":1}
I20260926 19:32:05.987489 21481 heartbeater.cc:347] Connected to a master server at 127.1.39.62:37119
I20260926 19:32:05.987803 21481 heartbeater.cc:464] Registering TS with master...
I20260926 19:32:05.988500 21481 heartbeater.cc:511] Master 127.1.39.62:37119 requested a full tablet report, sending...
I20260926 19:32:05.989938 21034 ts_manager.cc:194] Registered new tserver with Master: a74759bed29143d9939c48d33c7881c2 (127.1.39.3:36761)
I20260926 19:32:05.990742 21034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.39.3:40757
I20260926 19:32:06.002063 1180 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 19:32:06.016182 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.008473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.1.39.62:37119)
0926 19:32:06.008754 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.008763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.009081 (+ 318us) server_negotiation.cc:407] Connection header received
0926 19:32:06.009225 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.009232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.009377 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.009527 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:06.009966 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.011430 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.012863 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.013914 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.014084 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.015221 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:06.015232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:06.015241 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:06.015271 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:06.015322 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:06.015343 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:06.015469 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.015553 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.015676 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:32:06.021476 21034 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 19:32:06.037106 21034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 19:32:06.046617 21220 heartbeater.cc:503] Master 127.1.39.62:37119 was elected leader, sending a full tablet report...
I20260926 19:32:06.069623 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.053968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:46231 (local address 127.0.0.1:40744)
0926 19:32:06.060642 (+ 6674us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:06.060665 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:06.060785 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:06.061662 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:06.061668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:06.061697 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:06.062079 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.062091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.063997 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.064002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.066436 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.066451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.068619 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.068626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.068841 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.068889 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 19:32:06.069012 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 19:32:06.069197 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6526,"mutex_wait_us":1,"thread_start_us":3521,"threads_started":1}
I20260926 19:32:06.071173 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.054546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.1.39.2:46231)
0926 19:32:06.059119 (+ 4573us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.059134 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.060911 (+ 1777us) server_negotiation.cc:407] Connection header received
0926 19:32:06.061010 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.061017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.061227 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.061423 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:06.062252 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.063843 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.066665 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.068115 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.068236 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.068478 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.070641 (+ 2163us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.070813 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4336,"thread_start_us":4198,"threads_started":1}
I20260926 19:32:06.071185 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.052810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.1.39.3:36761)
0926 19:32:06.060374 (+ 7564us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.060384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.060428 (+ 44us) server_negotiation.cc:407] Connection header received
0926 19:32:06.060528 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.060537 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.060825 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.061010 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:06.062018 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.064636 (+ 2618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.068069 (+ 3433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.069482 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.069617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.069877 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.070650 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.070812 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7386,"thread_start_us":7230,"threads_started":1}
I20260926 19:32:06.072731 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.052511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.3:36761 (local address 127.0.0.1:55180)
0926 19:32:06.058098 (+ 5587us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:06.058164 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:06.058406 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:06.061346 (+ 2940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:06.061357 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:06.061393 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:06.061760 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.061774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.064790 (+ 3016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.064795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.067851 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.067865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.070011 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.070019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.070262 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.070293 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 19:32:06.070367 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 19:32:06.070497 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5196,"thread_start_us":7843,"threads_started":1}
I20260926 19:32:06.075419 21286 tablet_service.cc:1467] Processing CreateTablet for tablet 576424acd74944798bd43d0dba4c08fc (DEFAULT_TABLE table=test-table [id=ae8395fc076b4b26839ded3eedccf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:06.076983 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576424acd74944798bd43d0dba4c08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:06.077679 21416 tablet_service.cc:1467] Processing CreateTablet for tablet 576424acd74944798bd43d0dba4c08fc (DEFAULT_TABLE table=test-table [id=ae8395fc076b4b26839ded3eedccf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:06.079353 21416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576424acd74944798bd43d0dba4c08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:06.082556 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.068027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34173 (local address 127.0.0.1:55778)
0926 19:32:06.070389 (+ 2362us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:06.070405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:06.070485 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:06.074904 (+ 4419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:06.074910 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:06.074933 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:06.075066 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.075088 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.076906 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.076911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.079645 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.079660 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.081768 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.081775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.081961 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.081997 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 19:32:06.082099 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 19:32:06.082238 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2254,"thread_start_us":5465,"threads_started":1}
I20260926 19:32:06.083317 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.061236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.1.39.1:34173)
0926 19:32:06.074062 (+ 12826us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.074082 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.074113 (+ 31us) server_negotiation.cc:407] Connection header received
0926 19:32:06.074218 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.074225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.074456 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.074643 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:06.075241 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.076737 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.079829 (+ 3092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.081275 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.081421 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.081676 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.082796 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.082963 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12654,"threads_started":1}
I20260926 19:32:06.088963 21155 tablet_service.cc:1467] Processing CreateTablet for tablet 576424acd74944798bd43d0dba4c08fc (DEFAULT_TABLE table=test-table [id=ae8395fc076b4b26839ded3eedccf5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 19:32:06.090809 21155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 576424acd74944798bd43d0dba4c08fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:06.110508 21499 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Bootstrap starting.
I20260926 19:32:06.114132 21501 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Bootstrap starting.
I20260926 19:32:06.115564 21499 tablet_bootstrap.cc:654] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:06.117489 21499 log.cc:824] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:06.118150 21500 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Bootstrap starting.
I20260926 19:32:06.118259 21501 tablet_bootstrap.cc:654] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:06.120375 21501 log.cc:824] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:06.123222 21500 tablet_bootstrap.cc:654] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:06.124948 21500 log.cc:824] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:06.127458 21501 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: No bootstrap required, opened a new log
I20260926 19:32:06.127810 21501 ts_tablet_manager.cc:1397] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260926 19:32:06.127996 21499 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: No bootstrap required, opened a new log
I20260926 19:32:06.128283 21499 ts_tablet_manager.cc:1397] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20260926 19:32:06.135985 21500 tablet_bootstrap.cc:492] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: No bootstrap required, opened a new log
I20260926 19:32:06.136381 21500 ts_tablet_manager.cc:1397] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20260926 19:32:06.137537 21501 raft_consensus.cc:348] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.138123 21501 raft_consensus.cc:374] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:06.138259 21501 raft_consensus.cc:729] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe40c4608ec4718a34300090a6eac0b, State: Initialized, Role: FOLLOWER
I20260926 19:32:06.138957 21501 consensus_queue.cc:260] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.143411 21499 raft_consensus.cc:348] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.143992 21499 raft_consensus.cc:374] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:06.144119 21499 raft_consensus.cc:729] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3c36c094e845e9a6809fa76aacc6ae, State: Initialized, Role: FOLLOWER
I20260926 19:32:06.145056 21499 consensus_queue.cc:260] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.147186 21501 ts_tablet_manager.cc:1428] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260926 19:32:06.149657 21500 raft_consensus.cc:348] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.150297 21500 raft_consensus.cc:374] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:06.150444 21500 raft_consensus.cc:729] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74759bed29143d9939c48d33c7881c2, State: Initialized, Role: FOLLOWER
I20260926 19:32:06.151229 21500 consensus_queue.cc:260] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.151783 21499 ts_tablet_manager.cc:1428] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Time spent starting tablet: real 0.023s user 0.019s sys 0.003s
I20260926 19:32:06.151908 21351 heartbeater.cc:503] Master 127.1.39.62:37119 was elected leader, sending a full tablet report...
I20260926 19:32:06.152416 21481 heartbeater.cc:503] Master 127.1.39.62:37119 was elected leader, sending a full tablet report...
I20260926 19:32:06.153342 21500 ts_tablet_manager.cc:1428] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260926 19:32:06.168467 21507 raft_consensus.cc:482] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 19:32:06.168809 21507 raft_consensus.cc:504] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.170596 21507 leader_election.cc:288] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfe40c4608ec4718a34300090a6eac0b (127.1.39.1:34173), da3c36c094e845e9a6809fa76aacc6ae (127.1.39.2:46231)
I20260926 19:32:06.176952 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.171097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:42243 (local address 127.1.39.1:34173)
0926 19:32:06.171301 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.171306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.171719 (+ 413us) server_negotiation.cc:407] Connection header received
0926 19:32:06.172021 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.172031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.172146 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.172273 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:06.172829 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.173664 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.175249 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.176055 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.176190 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.176318 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.176573 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.176698 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 19:32:06.176995 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.171421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.1:34173 (local address 127.1.39.3:42243)
0926 19:32:06.171609 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:06.171625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:06.171791 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:06.172358 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:06.172364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:06.172400 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:06.172685 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.172694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.173790 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.173799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.175094 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.175106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.176218 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.176223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.176400 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.176435 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 19:32:06.176534 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 19:32:06.176683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"spinlock_wait_cycles":22400}
I20260926 19:32:06.178099 21175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576424acd74944798bd43d0dba4c08fc" candidate_uuid: "a74759bed29143d9939c48d33c7881c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe40c4608ec4718a34300090a6eac0b" is_pre_election: true
I20260926 19:32:06.178699 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.171211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.39.3:49147 (local address 127.1.39.2:46231)
0926 19:32:06.171389 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.171397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.172433 (+ 1036us) server_negotiation.cc:407] Connection header received
0926 19:32:06.172657 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.172662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.172744 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.172871 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 19:32:06.173602 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.174354 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.176372 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.177523 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.177724 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.177838 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.178231 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.178378 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:32:06.178756 21175 raft_consensus.cc:2454] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74759bed29143d9939c48d33c7881c2 in term 0.
I20260926 19:32:06.179313 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.171460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.2:46231 (local address 127.1.39.3:49147)
0926 19:32:06.172340 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:06.172357 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:06.172502 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:06.172887 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:06.172893 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:06.172922 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 19:32:06.173413 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.173424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.174528 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.174537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.176141 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:06.176155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.177760 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.177768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:06.177991 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.178032 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 19:32:06.178189 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 19:32:06.178342 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":48,"thread_start_us":470,"threads_started":1}
I20260926 19:32:06.179581 21369 leader_election.cc:302] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74759bed29143d9939c48d33c7881c2, bfe40c4608ec4718a34300090a6eac0b; no voters:
I20260926 19:32:06.180022 21507 raft_consensus.cc:2781] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 19:32:06.180181 21507 raft_consensus.cc:482] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 19:32:06.180305 21507 raft_consensus.cc:3037] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:06.180457 21306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576424acd74944798bd43d0dba4c08fc" candidate_uuid: "a74759bed29143d9939c48d33c7881c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c36c094e845e9a6809fa76aacc6ae" is_pre_election: true
I20260926 19:32:06.181005 21306 raft_consensus.cc:2454] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a74759bed29143d9939c48d33c7881c2 in term 0.
I20260926 19:32:06.182819 21507 raft_consensus.cc:504] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.183454 21507 leader_election.cc:288] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [CANDIDATE]: Term 1 election: Requested vote from peers bfe40c4608ec4718a34300090a6eac0b (127.1.39.1:34173), da3c36c094e845e9a6809fa76aacc6ae (127.1.39.2:46231)
I20260926 19:32:06.183950 21175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576424acd74944798bd43d0dba4c08fc" candidate_uuid: "a74759bed29143d9939c48d33c7881c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe40c4608ec4718a34300090a6eac0b"
I20260926 19:32:06.184018 21306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "576424acd74944798bd43d0dba4c08fc" candidate_uuid: "a74759bed29143d9939c48d33c7881c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da3c36c094e845e9a6809fa76aacc6ae"
I20260926 19:32:06.184156 21175 raft_consensus.cc:3037] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:06.184248 21306 raft_consensus.cc:3037] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:06.186496 21175 raft_consensus.cc:2454] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74759bed29143d9939c48d33c7881c2 in term 1.
I20260926 19:32:06.186571 21306 raft_consensus.cc:2454] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a74759bed29143d9939c48d33c7881c2 in term 1.
I20260926 19:32:06.186950 21369 leader_election.cc:302] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a74759bed29143d9939c48d33c7881c2, bfe40c4608ec4718a34300090a6eac0b; no voters:
I20260926 19:32:06.187253 21507 raft_consensus.cc:2781] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:06.187711 21507 raft_consensus.cc:686] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 1 LEADER]: Becoming Leader. State: Replica: a74759bed29143d9939c48d33c7881c2, State: Running, Role: LEADER
I20260926 19:32:06.188199 21507 consensus_queue.cc:237] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } }
I20260926 19:32:06.193483 21032 catalog_manager.cc:5495] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a74759bed29143d9939c48d33c7881c2 (127.1.39.3). New cstate: current_term: 1 leader_uuid: "a74759bed29143d9939c48d33c7881c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a74759bed29143d9939c48d33c7881c2" member_type: VOTER last_known_addr { host: "127.1.39.3" port: 36761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 } health_report { overall_health: UNKNOWN } } }
W20260926 19:32:06.221561 21482 tablet.cc:2369] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:06.221983 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.216166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.1.39.3:36761)
0926 19:32:06.216397 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.216402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.216421 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:32:06.216623 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.216627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.216712 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.216842 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:06.217187 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.218060 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.219600 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.220069 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.220172 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.220719 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:06.221339 (+ 620us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:06.221514 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.221579 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.221694 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 19:32:06.230343 21306 raft_consensus.cc:1261] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 1 FOLLOWER]: Refusing update from remote peer a74759bed29143d9939c48d33c7881c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:06.230347 21175 raft_consensus.cc:1261] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 1 FOLLOWER]: Refusing update from remote peer a74759bed29143d9939c48d33c7881c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 19:32:06.231349 21352 tablet.cc:2369] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:06.231384 21509 consensus_queue.cc:1035] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da3c36c094e845e9a6809fa76aacc6ae" member_type: VOTER last_known_addr { host: "127.1.39.2" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:06.231798 21507 consensus_queue.cc:1035] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe40c4608ec4718a34300090a6eac0b" member_type: VOTER last_known_addr { host: "127.1.39.1" port: 34173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:06.243409 21512 mvcc.cc:205] Tried to move back new op lower bound from 7333687813028888576 to 7333687812870774784. Current Snapshot: MvccSnapshot[applied={T|T < 7333687813028888576}]
I20260926 19:32:06.257179 21032 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:32:06.257529 21032 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 19:32:06.260869 21032 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff2d4479a75e437495f56034f512eb56: Sending DeleteTablet for 3 replicas of tablet 576424acd74944798bd43d0dba4c08fc
I20260926 19:32:06.262039 21416 tablet_service.cc:1514] Processing DeleteTablet for tablet 576424acd74944798bd43d0dba4c08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:06 UTC) from {username='slave'} at 127.0.0.1:55180
I20260926 19:32:06.262141 21155 tablet_service.cc:1514] Processing DeleteTablet for tablet 576424acd74944798bd43d0dba4c08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:06 UTC) from {username='slave'} at 127.0.0.1:55778
I20260926 19:32:06.262167 21286 tablet_service.cc:1514] Processing DeleteTablet for tablet 576424acd74944798bd43d0dba4c08fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 19:32:06 UTC) from {username='slave'} at 127.0.0.1:40744
I20260926 19:32:06.265209 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21093
I20260926 19:32:06.275098 21523 tablet_replica.cc:330] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: stopping tablet replica
I20260926 19:32:06.275389 21522 tablet_replica.cc:330] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: stopping tablet replica
I20260926 19:32:06.275750 21523 raft_consensus.cc:2229] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:32:06.275768 21522 raft_consensus.cc:2229] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 19:32:06.276331 21523 raft_consensus.cc:2258] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:06.276603 21522 raft_consensus.cc:2258] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:06.278309 21524 tablet_replica.cc:330] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: stopping tablet replica
I20260926 19:32:06.278775 21523 ts_tablet_manager.cc:1905] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:32:06.278743 21524 raft_consensus.cc:2229] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 19:32:06.279290 21524 raft_consensus.cc:2258] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 19:32:06.280939 21522 ts_tablet_manager.cc:1905] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:32:06.285157 21524 ts_tablet_manager.cc:1905] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 19:32:06.286419 21523 ts_tablet_manager.cc:1918] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:06.286624 21523 log.cc:1196] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-0/wal/wals/576424acd74944798bd43d0dba4c08fc
I20260926 19:32:06.287207 21523 ts_tablet_manager.cc:1939] T 576424acd74944798bd43d0dba4c08fc P bfe40c4608ec4718a34300090a6eac0b: Deleting consensus metadata
I20260926 19:32:06.288671 21522 ts_tablet_manager.cc:1918] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:06.288872 21522 log.cc:1196] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-2/wal/wals/576424acd74944798bd43d0dba4c08fc
I20260926 19:32:06.289186 21018 catalog_manager.cc:4841] TS bfe40c4608ec4718a34300090a6eac0b (127.1.39.1:34173): tablet 576424acd74944798bd43d0dba4c08fc (table test-table [id=ae8395fc076b4b26839ded3eedccf5f8]) successfully deleted
I20260926 19:32:06.289582 21522 ts_tablet_manager.cc:1939] T 576424acd74944798bd43d0dba4c08fc P a74759bed29143d9939c48d33c7881c2: Deleting consensus metadata
I20260926 19:32:06.291304 21018 catalog_manager.cc:4841] TS a74759bed29143d9939c48d33c7881c2 (127.1.39.3:36761): tablet 576424acd74944798bd43d0dba4c08fc (table test-table [id=ae8395fc076b4b26839ded3eedccf5f8]) successfully deleted
I20260926 19:32:06.292430 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.266587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55782 (local address 127.1.39.1:34173)
0926 19:32:06.266768 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.266775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.277651 (+ 10876us) server_negotiation.cc:407] Connection header received
0926 19:32:06.277855 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.277861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.277961 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.278067 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:06.278693 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.279818 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.281785 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.282451 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.282555 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.290811 (+ 8256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:06.290825 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:06.290843 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:06.291614 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:06.291663 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:06.291683 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:06.291823 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.291965 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.292068 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:32:06.292654 21524 ts_tablet_manager.cc:1918] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 19:32:06.292877 21524 log.cc:1196] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Deleting WAL directory at /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790450801920901-1180-0/minicluster-data/ts-1/wal/wals/576424acd74944798bd43d0dba4c08fc
I20260926 19:32:06.293154 21215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:06.293413 21524 ts_tablet_manager.cc:1939] T 576424acd74944798bd43d0dba4c08fc P da3c36c094e845e9a6809fa76aacc6ae: Deleting consensus metadata
I20260926 19:32:06.295249 21021 catalog_manager.cc:4841] TS da3c36c094e845e9a6809fa76aacc6ae (127.1.39.2:46231): tablet 576424acd74944798bd43d0dba4c08fc (table test-table [id=ae8395fc076b4b26839ded3eedccf5f8]) successfully deleted
I20260926 19:32:06.384716 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21093
I20260926 19:32:06.410004 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21224
I20260926 19:32:06.417558 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.410997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.1.39.2:46231)
0926 19:32:06.411198 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.411204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.411224 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:32:06.411351 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.411356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.411466 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.411588 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:06.412002 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.412806 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.414375 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.414800 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.414938 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.416081 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:06.416090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:06.416107 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:06.416695 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:06.416741 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:06.416765 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:06.416888 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.417027 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.417136 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:32:06.418360 21346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:06.512099 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21224
I20260926 19:32:06.536461 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21354
I20260926 19:32:06.544811 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:06.537449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.1.39.3:36761)
0926 19:32:06.537647 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:06.537652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:06.537721 (+ 69us) server_negotiation.cc:407] Connection header received
0926 19:32:06.537900 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:06.537905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:06.537984 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:06.538087 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:06.538552 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.539375 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.541198 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:06.541771 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:06.541869 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:06.543043 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:06.543052 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:06.543073 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:06.543668 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:06.543757 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:06.543786 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:06.543902 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:06.544149 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 19:32:06.544328 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 19:32:06.553270 21476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:06.674383 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21354
I20260926 19:32:06.694579 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21003
I20260926 19:32:06.695616 21064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:06.806041 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21003
2026-09-26T19:32:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2889 ms)
[----------] 6 tests from EncryptionPolicyTest (22384 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:06 dist-test-slave-9wb0 krb5kdc[21532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:06 dist-test-slave-9wb0 krb5kdc[21532](info): set up 2 sockets
Sep 26 19:32:06 dist-test-slave-9wb0 krb5kdc[21532](info): commencing operation
krb5kdc: starting...
W20260926 19:32:07.929904 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:08 dist-test-slave-9wb0 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:08.212350 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:08.554906 21545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:08.555243 21545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:08.555414 21545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:08.562522 21545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:08.562674 21545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:08.562781 21545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:08.562889 21545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:08.563019 21545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:08.572952 21545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43339
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:08.574312 21545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:08.575646 21545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:08.587735 21553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:08.587836 21551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:08.587786 21550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:08.588238 21545 server_base.cc:1034] running on GCE node
I20260926 19:32:08.589185 21545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:08.590607 21545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:08.591835 21545 hybrid_clock.cc:648] HybridClock initialized: now 1790451128591788 us; error 54 us; skew 500 ppm
I20260926 19:32:08.592171 21545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:08 dist-test-slave-9wb0 krb5kdc[21532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:08.595234 21545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:08.596761 21545 webserver.cc:460] Webserver started at http://127.0.0.1:39577/ using document root <none> and password file <none>
I20260926 19:32:08.597535 21545 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:08.597671 21545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:08.597961 21545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:08.600872 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "a4a685dc1c954b3ca271de89c5c16c55"
format_stamp: "Formatted at 2026-09-26 19:32:08 on dist-test-slave-9wb0"
server_key: "401aef9ab92d6363a542d13a43a25d82"
server_key_iv: "f16c134a3ca369c83c741e83cb83a208"
server_key_version: "encryptionkey@0"
I20260926 19:32:08.601699 21545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "a4a685dc1c954b3ca271de89c5c16c55"
format_stamp: "Formatted at 2026-09-26 19:32:08 on dist-test-slave-9wb0"
server_key: "401aef9ab92d6363a542d13a43a25d82"
server_key_iv: "f16c134a3ca369c83c741e83cb83a208"
server_key_version: "encryptionkey@0"
I20260926 19:32:08.606601 21545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:32:08.609905 21560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:08.611109 21545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:08.611284 21545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "a4a685dc1c954b3ca271de89c5c16c55"
format_stamp: "Formatted at 2026-09-26 19:32:08 on dist-test-slave-9wb0"
server_key: "401aef9ab92d6363a542d13a43a25d82"
server_key_iv: "f16c134a3ca369c83c741e83cb83a208"
server_key_version: "encryptionkey@0"
I20260926 19:32:08.611479 21545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:08.661662 21545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:08.666262 21545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:08.666596 21545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:08.685585 21545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43339
I20260926 19:32:08.685618 21611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43339 every 8 connection(s)
I20260926 19:32:08.687038 21545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:08.690512 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21545
I20260926 19:32:08.690757 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:08.691049 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:08.691546 1180 external_mini_cluster.cc:1428] Setting key 6a30c5b0930749498f68fb10698877a8
I20260926 19:32:08.705343 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55: Bootstrap starting.
Sep 26 19:32:08 dist-test-slave-9wb0 krb5kdc[21532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:32:08.708832 21612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:08.709997 21612 log.cc:824] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:08.713115 21612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55: No bootstrap required, opened a new log
I20260926 19:32:08.717722 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:08.693699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47573 (local address 127.0.0.1:43339)
0926 19:32:08.695893 (+ 2194us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:08.695918 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:08.696307 (+ 389us) server_negotiation.cc:407] Connection header received
0926 19:32:08.698689 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:08.698719 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:08.699199 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:08.699651 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:08.701245 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:08.702678 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:08.704073 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:08.704772 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:08.704973 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:08.709487 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:08.709495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:08.709511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:08.709562 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:08.712695 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:08.713662 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:08.713669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:08.713675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:08.713839 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:08.714313 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:08.714335 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:08.714341 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:08.714920 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:08.715268 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:08.715851 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 19:32:08.716298 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1904,"thread_start_us":1602,"threads_started":1}
I20260926 19:32:08.721917 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:08.722844 21612 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260926 19:32:08.723212 21612 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:08.723330 21612 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a685dc1c954b3ca271de89c5c16c55, State: Initialized, Role: FOLLOWER
I20260926 19:32:08.723898 21612 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260926 19:32:08.724133 21612 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:08.724259 21612 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:08.724432 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:08.726960 21612 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260926 19:32:08.727545 21612 leader_election.cc:302] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4a685dc1c954b3ca271de89c5c16c55; no voters:
I20260926 19:32:08.728224 21612 leader_election.cc:288] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:08.728353 21622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:08.729583 21622 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [term 1 LEADER]: Becoming Leader. State: Replica: a4a685dc1c954b3ca271de89c5c16c55, State: Running, Role: LEADER
I20260926 19:32:08.730209 21622 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } }
I20260926 19:32:08.730561 21612 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:08.734964 21623 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4a685dc1c954b3ca271de89c5c16c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } } }
I20260926 19:32:08.735383 21623 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:08.735881 21624 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4a685dc1c954b3ca271de89c5c16c55. Latest consensus state: current_term: 1 leader_uuid: "a4a685dc1c954b3ca271de89c5c16c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a685dc1c954b3ca271de89c5c16c55" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43339 } } }
I20260926 19:32:08.736173 21624 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:08.747886 21629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:08.748807 21615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:08.727022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57155 (local address 127.0.0.1:43339)
0926 19:32:08.727214 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:08.727220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:08.727587 (+ 367us) server_negotiation.cc:407] Connection header received
0926 19:32:08.727826 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:08.727831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:08.727913 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:08.728054 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:08.730133 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:08.731366 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:08.733833 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:08.734539 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:08.734667 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:08.738958 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:08.738968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:08.738976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:08.739029 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:08.743697 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:08.745196 (+ 1499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:08.745205 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:08.745212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:08.745478 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:08.746247 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:08.746259 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:08.746265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:08.746922 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:08.747257 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:08.748284 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0926 19:32:08.748481 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:32:08.753360 21629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:08.762377 21629 catalog_manager.cc:1347] Generated new cluster ID: f11f7f3409274a17a214ba1534948e7f
I20260926 19:32:08.762465 21629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:08.778025 21629 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:08.779839 21629 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:08.788807 21629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4a685dc1c954b3ca271de89c5c16c55: Generated new TSK 0
I20260926 19:32:08.789867 21629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:08.795354 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21545
I20260926 19:32:08.796501 21607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:08.887183 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21545
W20260926 19:32:08.905833 21530 connection.cc:535] client connection to 127.0.0.1:43339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43339): Connection reset by peer (error 104)
2026-09-26T19:32:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2075 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:09 dist-test-slave-9wb0 krb5kdc[21645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:09 dist-test-slave-9wb0 krb5kdc[21645](info): set up 2 sockets
Sep 26 19:32:09 dist-test-slave-9wb0 krb5kdc[21645](info): commencing operation
krb5kdc: starting...
W20260926 19:32:10.030290 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:10 dist-test-slave-9wb0 krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:10.320765 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:10.657212 21658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:10.657641 21658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:10.657809 21658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:10.665115 21658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:10.665282 21658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:10.665402 21658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:10.665512 21658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:10.665597 21658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:10.675618 21658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:10.676980 21658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:10.678363 21658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:10.690388 21663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:10.690464 21664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:10.690464 21666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:10.691270 21658 server_base.cc:1034] running on GCE node
I20260926 19:32:10.692018 21658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:10.693512 21658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:10.694746 21658 hybrid_clock.cc:648] HybridClock initialized: now 1790451130694710 us; error 49 us; skew 500 ppm
I20260926 19:32:10.695101 21658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:10 dist-test-slave-9wb0 krb5kdc[21645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:10.698364 21658 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:10.699828 21658 webserver.cc:460] Webserver started at http://127.0.0.1:45717/ using document root <none> and password file <none>
I20260926 19:32:10.700538 21658 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:10.700692 21658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:10.701014 21658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:10.704043 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "50aa360b6515409b96af7f93e897ab24"
format_stamp: "Formatted at 2026-09-26 19:32:10 on dist-test-slave-9wb0"
server_key: "379fcc39608263eed47695ea2827d2c1"
server_key_iv: "3641233785b6fa546b83566fe9e1bc3c"
server_key_version: "encryptionkey@0"
I20260926 19:32:10.704809 21658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "50aa360b6515409b96af7f93e897ab24"
format_stamp: "Formatted at 2026-09-26 19:32:10 on dist-test-slave-9wb0"
server_key: "379fcc39608263eed47695ea2827d2c1"
server_key_iv: "3641233785b6fa546b83566fe9e1bc3c"
server_key_version: "encryptionkey@0"
I20260926 19:32:10.710054 21658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 19:32:10.713529 21673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:10.714744 21658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:32:10.714918 21658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "50aa360b6515409b96af7f93e897ab24"
format_stamp: "Formatted at 2026-09-26 19:32:10 on dist-test-slave-9wb0"
server_key: "379fcc39608263eed47695ea2827d2c1"
server_key_iv: "3641233785b6fa546b83566fe9e1bc3c"
server_key_version: "encryptionkey@0"
I20260926 19:32:10.715121 21658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:10.733996 21658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:10.738665 21658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:10.738989 21658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:10.758884 21658 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33325
I20260926 19:32:10.758919 21724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33325 every 8 connection(s)
I20260926 19:32:10.760303 21658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:10.764089 21725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:10.769583 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21658
I20260926 19:32:10.769768 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:10.770228 1180 external_mini_cluster.cc:1428] Setting key 1db5e6134aa849c4fe5cbfc0020df8eb
I20260926 19:32:10.775100 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24: Bootstrap starting.
I20260926 19:32:10.778869 21725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:10.780068 21725 log.cc:824] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:10.783143 21725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24: No bootstrap required, opened a new log
I20260926 19:32:10.783937 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:10.772038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32971 (local address 127.0.0.1:33325)
0926 19:32:10.772898 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:10.772909 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:10.772951 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:32:10.774647 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:10.774671 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:10.775058 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:10.775434 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:10.776065 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:10.777457 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:10.779451 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:10.780130 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:10.780319 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:10.781675 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:10.781683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:10.781697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:10.782230 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:10.782316 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:10.782340 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:10.782509 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:10.782780 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 19:32:10.782946 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":405,"threads_started":1}
I20260926 19:32:10.787770 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:10.791260 21725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } }
I20260926 19:32:10.791659 21725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:10.791795 21725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50aa360b6515409b96af7f93e897ab24, State: Initialized, Role: FOLLOWER
I20260926 19:32:10.792569 21725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } }
I20260926 19:32:10.792889 21725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:10.793068 21725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:10.793253 21725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:10.796087 21725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } }
I20260926 19:32:10.796747 21725 leader_election.cc:302] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50aa360b6515409b96af7f93e897ab24; no voters:
I20260926 19:32:10.797504 21725 leader_election.cc:288] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:10.797605 21735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:10.799146 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:10.792154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35671 (local address 127.0.0.1:33325)
0926 19:32:10.792356 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:10.792367 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:10.792788 (+ 421us) server_negotiation.cc:407] Connection header received
0926 19:32:10.792981 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:10.792992 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:10.793094 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:10.793207 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:10.793779 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:10.794674 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:10.795992 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:10.796589 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:10.796743 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:10.798007 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:10.798018 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:10.798025 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:10.798052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:10.798154 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:10.798189 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:10.798679 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:10.798761 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 19:32:10.798807 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:32:10.800423 21725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:10.801607 21735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [term 1 LEADER]: Becoming Leader. State: Replica: 50aa360b6515409b96af7f93e897ab24, State: Running, Role: LEADER
I20260926 19:32:10.802335 21735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } }
I20260926 19:32:10.806844 21736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50aa360b6515409b96af7f93e897ab24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } } }
I20260926 19:32:10.807183 21736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:10.806651 21737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50aa360b6515409b96af7f93e897ab24. Latest consensus state: current_term: 1 leader_uuid: "50aa360b6515409b96af7f93e897ab24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50aa360b6515409b96af7f93e897ab24" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33325 } } }
I20260926 19:32:10.807497 21737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:10.809122 21749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:10.814426 21749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:10.823087 21749 catalog_manager.cc:1347] Generated new cluster ID: 53a72da813624516975cfaa69ececdd4
I20260926 19:32:10.823210 21749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:10.841634 21749 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:10.843653 21749 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:10.858032 21749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50aa360b6515409b96af7f93e897ab24: Generated new TSK 0
I20260926 19:32:10.858932 21749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:10.873360 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21658
W20260926 19:32:10.874575 21720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32971
W20260926 19:32:10.875388 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21658: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:10.875571 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21658
W20260926 19:32:10.893848 21643 connection.cc:535] client connection to 127.0.0.1:33325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33325): Connection reset by peer (error 104)
2026-09-26T19:32:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:10 dist-test-slave-9wb0 krb5kdc[21756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:10 dist-test-slave-9wb0 krb5kdc[21756](info): set up 2 sockets
Sep 26 19:32:10 dist-test-slave-9wb0 krb5kdc[21756](info): commencing operation
krb5kdc: starting...
W20260926 19:32:12.005707 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:12 dist-test-slave-9wb0 krb5kdc[21756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:12.284235 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:12.616719 21769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:12.616999 21769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:12.617100 21769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:12.624405 21769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:12.624517 21769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:12.624563 21769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:12.624625 21769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:12.624662 21769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:12.634661 21769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:12.635896 21769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:12.637130 21769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:12.649035 21774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:12.649004 21775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:12.649089 21777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:12.649844 21769 server_base.cc:1034] running on GCE node
I20260926 19:32:12.650722 21769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:12.652084 21769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:12.653337 21769 hybrid_clock.cc:648] HybridClock initialized: now 1790451132653255 us; error 29 us; skew 500 ppm
I20260926 19:32:12.653697 21769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:12 dist-test-slave-9wb0 krb5kdc[21756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:12.661530 21769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:12.663041 21769 webserver.cc:460] Webserver started at http://127.0.0.1:45421/ using document root <none> and password file <none>
I20260926 19:32:12.663746 21769 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:12.663895 21769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:12.664217 21769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:12.667259 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "1fbbc48a88f044df9b73fff722abfca6"
format_stamp: "Formatted at 2026-09-26 19:32:12 on dist-test-slave-9wb0"
server_key: "5af5c4abac94462000efd9db8c123a7e"
server_key_iv: "01b7291b412bf890f9565710056295d0"
server_key_version: "encryptionkey@0"
I20260926 19:32:12.668015 21769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "1fbbc48a88f044df9b73fff722abfca6"
format_stamp: "Formatted at 2026-09-26 19:32:12 on dist-test-slave-9wb0"
server_key: "5af5c4abac94462000efd9db8c123a7e"
server_key_iv: "01b7291b412bf890f9565710056295d0"
server_key_version: "encryptionkey@0"
I20260926 19:32:12.673079 21769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:32:12.676421 21784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:12.677783 21769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:12.677970 21769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "1fbbc48a88f044df9b73fff722abfca6"
format_stamp: "Formatted at 2026-09-26 19:32:12 on dist-test-slave-9wb0"
server_key: "5af5c4abac94462000efd9db8c123a7e"
server_key_iv: "01b7291b412bf890f9565710056295d0"
server_key_version: "encryptionkey@0"
I20260926 19:32:12.678182 21769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:12.683873 21769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:12.687973 21769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:12.688277 21769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:12.707208 21769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260926 19:32:12.707250 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260926 19:32:12.708633 21769 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:12.710470 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21769
I20260926 19:32:12.710671 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:12.711248 1180 external_mini_cluster.cc:1428] Setting key 70dfee8186be6c0a2ac5f3f1a6381054
I20260926 19:32:12.713138 21836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:12.719043 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:12.713043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34961 (local address 127.0.0.1:35665)
0926 19:32:12.713893 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:12.713904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:12.713946 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:32:12.715505 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:12.715526 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:12.715855 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:12.716253 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:12.717343 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:12.717352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:12.717368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:12.717393 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:12.717473 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:12.717494 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:12.717731 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:12.717868 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 19:32:12.718043 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":346,"threads_started":1}
I20260926 19:32:12.722596 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:12.724882 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6: Bootstrap starting.
I20260926 19:32:12.728798 21836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:12.729606 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:12.726879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35589 (local address 127.0.0.1:35665)
0926 19:32:12.727031 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:12.727039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:12.727620 (+ 581us) server_negotiation.cc:407] Connection header received
0926 19:32:12.727811 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:12.727815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:12.727863 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:12.727964 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:12.728908 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:12.728916 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:12.728923 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:12.728944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:12.728989 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:12.729014 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:12.729136 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:12.729259 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 19:32:12.729342 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 19:32:12.730130 21836 log.cc:824] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:12.732947 21836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6: No bootstrap required, opened a new log
I20260926 19:32:12.741070 21836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 19:32:12.741562 21836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:12.741683 21836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbbc48a88f044df9b73fff722abfca6, State: Initialized, Role: FOLLOWER
I20260926 19:32:12.742262 21836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 19:32:12.742502 21836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:12.742647 21836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:12.742837 21836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:12.745258 21836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 19:32:12.745888 21836 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbbc48a88f044df9b73fff722abfca6; no voters:
I20260926 19:32:12.746493 21836 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:12.746701 21846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:12.747447 21846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [term 1 LEADER]: Becoming Leader. State: Replica: 1fbbc48a88f044df9b73fff722abfca6, State: Running, Role: LEADER
I20260926 19:32:12.747985 21846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260926 19:32:12.748626 21836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:12.751062 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbbc48a88f044df9b73fff722abfca6. Latest consensus state: current_term: 1 leader_uuid: "1fbbc48a88f044df9b73fff722abfca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260926 19:32:12.751574 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:12.752151 21847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbbc48a88f044df9b73fff722abfca6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbbc48a88f044df9b73fff722abfca6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260926 19:32:12.752487 21847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:12.754768 21853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:12.760313 21853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:12.769196 21853 catalog_manager.cc:1347] Generated new cluster ID: 0fceacd60796493895f23a656ab02369
I20260926 19:32:12.769337 21853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:12.783167 21853 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:12.783600 21853 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:12.799479 21853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbbc48a88f044df9b73fff722abfca6: Generated new TSK 0
I20260926 19:32:12.800340 21853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:12.812188 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21769
W20260926 19:32:12.814069 21831 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34961
W20260926 19:32:12.814818 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21769: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:12.814954 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21769
2026-09-26T19:32:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:12 dist-test-slave-9wb0 krb5kdc[21867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:12 dist-test-slave-9wb0 krb5kdc[21867](info): set up 2 sockets
Sep 26 19:32:12 dist-test-slave-9wb0 krb5kdc[21867](info): commencing operation
krb5kdc: starting...
W20260926 19:32:13.946185 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:14.228786 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:14.574391 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:14.574738 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:14.574898 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:14.582103 21880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:14.582267 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:14.582369 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:14.582474 21880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:14.582549 21880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:14.592409 21880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:14.593786 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:14.595113 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:14.606634 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:14.606719 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:14.606676 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:14.607581 21880 server_base.cc:1034] running on GCE node
I20260926 19:32:14.608359 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:14.609872 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:14.611110 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790451134611053 us; error 45 us; skew 500 ppm
I20260926 19:32:14.611471 21880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:14.614554 21880 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:14.616003 21880 webserver.cc:460] Webserver started at http://127.0.0.1:37511/ using document root <none> and password file <none>
I20260926 19:32:14.616663 21880 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:14.616781 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:14.617054 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:14.620045 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "6f2435d8f26c45ff806e7c9a1fc299ac"
format_stamp: "Formatted at 2026-09-26 19:32:14 on dist-test-slave-9wb0"
server_key: "078193a5e1d5f7932aab1d9e1facd7c2"
server_key_iv: "37e551a3e045a9a173fcd072056c2ff7"
server_key_version: "encryptionkey@0"
I20260926 19:32:14.620744 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "6f2435d8f26c45ff806e7c9a1fc299ac"
format_stamp: "Formatted at 2026-09-26 19:32:14 on dist-test-slave-9wb0"
server_key: "078193a5e1d5f7932aab1d9e1facd7c2"
server_key_iv: "37e551a3e045a9a173fcd072056c2ff7"
server_key_version: "encryptionkey@0"
I20260926 19:32:14.625859 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260926 19:32:14.629155 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:14.630499 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:14.630687 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "6f2435d8f26c45ff806e7c9a1fc299ac"
format_stamp: "Formatted at 2026-09-26 19:32:14 on dist-test-slave-9wb0"
server_key: "078193a5e1d5f7932aab1d9e1facd7c2"
server_key_iv: "37e551a3e045a9a173fcd072056c2ff7"
server_key_version: "encryptionkey@0"
I20260926 19:32:14.630877 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:14.660794 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:14.664896 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:14.665203 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:14.683722 21880 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42815
I20260926 19:32:14.683750 21946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42815 every 8 connection(s)
I20260926 19:32:14.685110 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:14.686631 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21880
I20260926 19:32:14.686805 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:14.687266 1180 external_mini_cluster.cc:1428] Setting key 2dabb98fcbffddb9008137b43586fde8
I20260926 19:32:14.689406 21947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:32:14.702472 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac: Bootstrap starting.
I20260926 19:32:14.705607 21947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:14.706622 21947 log.cc:824] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:14.709508 21947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac: No bootstrap required, opened a new log
I20260926 19:32:14.709834 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:14.688971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41425 (local address 127.0.0.1:42815)
0926 19:32:14.689805 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:14.689817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:14.689856 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:32:14.691463 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:14.691490 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:14.691872 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:14.692186 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:14.693952 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:14.695294 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:14.697116 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:14.697819 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:14.697941 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:14.702751 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:14.702758 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:14.702772 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:14.702807 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:14.705926 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:14.706583 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:14.706595 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:14.706604 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:14.706786 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:14.707177 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:14.707183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:14.707186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:14.707700 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:14.707997 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:14.708418 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 19:32:14.708829 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":397,"threads_started":1}
I20260926 19:32:14.713459 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:14.717795 21947 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260926 19:32:14.718290 21947 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:14.718433 21947 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f2435d8f26c45ff806e7c9a1fc299ac, State: Initialized, Role: FOLLOWER
I20260926 19:32:14.719151 21947 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260926 19:32:14.719450 21947 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:14.719594 21947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:14.719798 21947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:14.722806 21947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260926 19:32:14.723414 21947 leader_election.cc:302] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f2435d8f26c45ff806e7c9a1fc299ac; no voters:
I20260926 19:32:14.724103 21947 leader_election.cc:288] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:14.724207 21957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:14.726867 21947 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:14.728132 21957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [term 1 LEADER]: Becoming Leader. State: Replica: 6f2435d8f26c45ff806e7c9a1fc299ac, State: Running, Role: LEADER
I20260926 19:32:14.729657 21957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260926 19:32:14.732981 21959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f2435d8f26c45ff806e7c9a1fc299ac. Latest consensus state: current_term: 1 leader_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260926 19:32:14.733484 21959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:14.734054 21958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f2435d8f26c45ff806e7c9a1fc299ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260926 19:32:14.734364 21958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:14.735306 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:14.717640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38463 (local address 127.0.0.1:42815)
0926 19:32:14.717800 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:14.717821 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:14.718229 (+ 408us) server_negotiation.cc:407] Connection header received
0926 19:32:14.718430 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:14.718435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:14.718510 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:14.718615 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:14.720584 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:14.721450 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:14.723212 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:14.723720 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:14.723814 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:14.727174 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:14.727181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:14.727185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:14.727223 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:14.730935 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:14.731902 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:14.731910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:14.731914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:14.732183 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:14.732631 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:14.732636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:14.732640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:14.733089 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:14.733405 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:14.734638 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0926 19:32:14.734873 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 19:32:14.737562 21968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:14.741760 21968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:14.750228 21968 catalog_manager.cc:1347] Generated new cluster ID: 83a341a6298642f8ad6050248065da19
I20260926 19:32:14.750349 21968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:14.776546 21968 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:14.777971 21968 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:14.786404 21968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f2435d8f26c45ff806e7c9a1fc299ac: Generated new TSK 0
I20260926 19:32:14.787360 21968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:14.792750 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21880
I20260926 19:32:14.794052 21942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:14.883909 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21880
2026-09-26T19:32:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21979](info): set up 2 sockets
Sep 26 19:32:14 dist-test-slave-9wb0 krb5kdc[21979](info): commencing operation
krb5kdc: starting...
W20260926 19:32:15.993916 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:16 dist-test-slave-9wb0 krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:16.274907 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:16.614068 21992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:16.614500 21992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:16.614704 21992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:16.621923 21992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:16.622076 21992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:16.622161 21992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:16.622258 21992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:16.622346 21992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:16.632293 21992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:16.633648 21992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:16.634970 21992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:16.652199 21998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:16.652310 21997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:16.652312 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:16.653172 21992 server_base.cc:1034] running on GCE node
I20260926 19:32:16.654081 21992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:16.655560 21992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:16.656778 21992 hybrid_clock.cc:648] HybridClock initialized: now 1790451136656736 us; error 28 us; skew 500 ppm
I20260926 19:32:16.657092 21992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:16 dist-test-slave-9wb0 krb5kdc[21979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:16.660755 21992 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:16.662384 21992 webserver.cc:460] Webserver started at http://127.0.0.1:40733/ using document root <none> and password file <none>
I20260926 19:32:16.663103 21992 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:16.663235 21992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:16.663558 21992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:16.666594 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "f7e864fc6ab34f718c9404ce1fa25580"
format_stamp: "Formatted at 2026-09-26 19:32:16 on dist-test-slave-9wb0"
server_key: "778d4d50d203a63181c34bc4635444a3"
server_key_iv: "86f5be1922811d95c6da642be387a1b4"
server_key_version: "encryptionkey@0"
I20260926 19:32:16.667330 21992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "f7e864fc6ab34f718c9404ce1fa25580"
format_stamp: "Formatted at 2026-09-26 19:32:16 on dist-test-slave-9wb0"
server_key: "778d4d50d203a63181c34bc4635444a3"
server_key_iv: "86f5be1922811d95c6da642be387a1b4"
server_key_version: "encryptionkey@0"
I20260926 19:32:16.672344 21992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:32:16.675832 22007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:16.677165 21992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:32:16.677393 21992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "f7e864fc6ab34f718c9404ce1fa25580"
format_stamp: "Formatted at 2026-09-26 19:32:16 on dist-test-slave-9wb0"
server_key: "778d4d50d203a63181c34bc4635444a3"
server_key_iv: "86f5be1922811d95c6da642be387a1b4"
server_key_version: "encryptionkey@0"
I20260926 19:32:16.677606 21992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:16.694633 21992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:16.698997 21992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:16.699331 21992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:16.718144 21992 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38121
I20260926 19:32:16.718194 22058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38121 every 8 connection(s)
I20260926 19:32:16.719491 21992 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:16.721926 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 21992
I20260926 19:32:16.722115 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:16.722676 1180 external_mini_cluster.cc:1428] Setting key 5da7677af8298c1babe961ee497e6e89
I20260926 19:32:16.723716 22059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:16.735702 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:16.724369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41327 (local address 127.0.0.1:38121)
0926 19:32:16.725150 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:16.725160 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:16.725199 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:32:16.726780 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:16.726801 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:16.727239 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:16.727559 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:16.728122 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:16.729485 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:16.731365 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:16.732032 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:16.732180 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:16.733456 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:16.733466 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:16.733479 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:16.734031 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:16.734124 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:16.734154 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:16.734250 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:16.734470 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 19:32:16.734752 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":366,"threads_started":1}
I20260926 19:32:16.737139 22059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580: Bootstrap starting.
I20260926 19:32:16.739428 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:16.740530 22059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:16.741564 22059 log.cc:824] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:16.744524 22059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580: No bootstrap required, opened a new log
I20260926 19:32:16.750083 22062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:16.743941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58727 (local address 127.0.0.1:38121)
0926 19:32:16.744138 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:16.744144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:16.744532 (+ 388us) server_negotiation.cc:407] Connection header received
0926 19:32:16.744706 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:16.744710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:16.744768 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:16.744869 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:16.745420 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:16.746173 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:16.747526 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:16.747984 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:16.748123 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:16.749259 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:16.749266 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:16.749270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:16.749328 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:16.749382 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:16.749406 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:16.749524 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:16.749671 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 19:32:16.749762 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:32:16.752619 22059 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260926 19:32:16.752993 22059 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:16.753098 22059 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7e864fc6ab34f718c9404ce1fa25580, State: Initialized, Role: FOLLOWER
I20260926 19:32:16.753731 22059 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260926 19:32:16.753973 22059 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:16.754124 22059 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:16.754300 22059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:16.756842 22059 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260926 19:32:16.757433 22059 leader_election.cc:302] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7e864fc6ab34f718c9404ce1fa25580; no voters:
I20260926 19:32:16.758024 22059 leader_election.cc:288] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:16.758172 22069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:16.759145 22069 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [term 1 LEADER]: Becoming Leader. State: Replica: f7e864fc6ab34f718c9404ce1fa25580, State: Running, Role: LEADER
I20260926 19:32:16.759776 22069 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } }
I20260926 19:32:16.760327 22059 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:16.762802 22071 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7e864fc6ab34f718c9404ce1fa25580. Latest consensus state: current_term: 1 leader_uuid: "f7e864fc6ab34f718c9404ce1fa25580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
I20260926 19:32:16.762858 22070 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7e864fc6ab34f718c9404ce1fa25580" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7e864fc6ab34f718c9404ce1fa25580" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38121 } } }
I20260926 19:32:16.763214 22071 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:16.763279 22070 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:16.766379 22075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:16.771704 22075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:16.780859 22075 catalog_manager.cc:1347] Generated new cluster ID: ccf409406ba3418c97d3ecb1fafc31db
I20260926 19:32:16.781008 22075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:16.811321 22075 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:16.812656 22075 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:16.819825 22075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7e864fc6ab34f718c9404ce1fa25580: Generated new TSK 0
I20260926 19:32:16.820647 22075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:16.834035 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21992
W20260926 19:32:16.835238 22054 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41327
W20260926 19:32:16.836021 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 21992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:16.836176 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 21992
2026-09-26T19:32:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:16 dist-test-slave-9wb0 krb5kdc[22090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:16 dist-test-slave-9wb0 krb5kdc[22090](info): set up 2 sockets
Sep 26 19:32:16 dist-test-slave-9wb0 krb5kdc[22090](info): commencing operation
krb5kdc: starting...
W20260926 19:32:17.949725 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:18 dist-test-slave-9wb0 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:18.224618 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:18.561259 22103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:18.561581 22103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:18.561764 22103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:18.569077 22103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:18.569214 22103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:18.569355 22103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:18.569456 22103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:18.569542 22103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:18.579341 22103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:18.580641 22103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:18.581948 22103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:18.593400 22108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:18.593482 22111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:18.593441 22109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:18.594578 22103 server_base.cc:1034] running on GCE node
I20260926 19:32:18.595325 22103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:18.596696 22103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:18.597926 22103 hybrid_clock.cc:648] HybridClock initialized: now 1790451138597876 us; error 58 us; skew 500 ppm
I20260926 19:32:18.598294 22103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:18 dist-test-slave-9wb0 krb5kdc[22090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:18.607859 22103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:18.609407 22103 webserver.cc:460] Webserver started at http://127.0.0.1:44717/ using document root <none> and password file <none>
I20260926 19:32:18.610076 22103 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:18.610176 22103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:18.610436 22103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:18.613343 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "04671e11e6d440e1b13ba01896990675"
format_stamp: "Formatted at 2026-09-26 19:32:18 on dist-test-slave-9wb0"
server_key: "78752f757ad5031d0bf1613406c21dde"
server_key_iv: "eb0744135f4b8fe3bd44838b1220568f"
server_key_version: "encryptionkey@0"
I20260926 19:32:18.614069 22103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "04671e11e6d440e1b13ba01896990675"
format_stamp: "Formatted at 2026-09-26 19:32:18 on dist-test-slave-9wb0"
server_key: "78752f757ad5031d0bf1613406c21dde"
server_key_iv: "eb0744135f4b8fe3bd44838b1220568f"
server_key_version: "encryptionkey@0"
I20260926 19:32:18.619077 22103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260926 19:32:18.622417 22118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:18.623660 22103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260926 19:32:18.623795 22103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "04671e11e6d440e1b13ba01896990675"
format_stamp: "Formatted at 2026-09-26 19:32:18 on dist-test-slave-9wb0"
server_key: "78752f757ad5031d0bf1613406c21dde"
server_key_iv: "eb0744135f4b8fe3bd44838b1220568f"
server_key_version: "encryptionkey@0"
I20260926 19:32:18.623946 22103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:18.629227 22103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:18.633016 22103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:18.633263 22103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:18.651255 22103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45683
I20260926 19:32:18.651336 22169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45683 every 8 connection(s)
I20260926 19:32:18.652580 22103 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:18.656157 22170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:18.660435 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22103
I20260926 19:32:18.660631 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:18.661058 1180 external_mini_cluster.cc:1428] Setting key 525f055f50ff293721db4b1e2ce837f4
I20260926 19:32:18.667130 22170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675: Bootstrap starting.
I20260926 19:32:18.668768 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:18.662778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48983 (local address 127.0.0.1:45683)
0926 19:32:18.663615 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:18.663632 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:18.663684 (+ 52us) server_negotiation.cc:407] Connection header received
0926 19:32:18.665615 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:18.665636 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:18.665941 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:18.666243 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:18.667330 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:18.667335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:18.667347 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:18.667361 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:18.667421 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:18.667445 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:18.667613 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:18.667722 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 19:32:18.667904 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":369,"threads_started":1}
I20260926 19:32:18.670388 22170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:18.671586 22170 log.cc:824] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:18.672497 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:18.674767 22170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675: No bootstrap required, opened a new log
I20260926 19:32:18.679939 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:18.677574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58555 (local address 127.0.0.1:45683)
0926 19:32:18.677771 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:18.677777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:18.678083 (+ 306us) server_negotiation.cc:407] Connection header received
0926 19:32:18.678274 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:18.678277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:18.678322 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:18.678402 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:18.679314 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:18.679318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:18.679321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:18.679335 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:18.679373 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:18.679387 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:18.679470 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:18.679599 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 19:32:18.679642 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 19:32:18.683812 22170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } }
I20260926 19:32:18.684181 22170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:18.684288 22170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04671e11e6d440e1b13ba01896990675, State: Initialized, Role: FOLLOWER
I20260926 19:32:18.684847 22170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } }
I20260926 19:32:18.685092 22170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:18.685232 22170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:18.685446 22170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:18.687572 22170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } }
I20260926 19:32:18.688055 22170 leader_election.cc:302] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04671e11e6d440e1b13ba01896990675; no voters:
I20260926 19:32:18.688614 22170 leader_election.cc:288] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:18.688735 22180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:18.689492 22180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [term 1 LEADER]: Becoming Leader. State: Replica: 04671e11e6d440e1b13ba01896990675, State: Running, Role: LEADER
I20260926 19:32:18.690032 22180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } }
I20260926 19:32:18.690780 22170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:18.692766 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04671e11e6d440e1b13ba01896990675. Latest consensus state: current_term: 1 leader_uuid: "04671e11e6d440e1b13ba01896990675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } } }
I20260926 19:32:18.692791 22181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04671e11e6d440e1b13ba01896990675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04671e11e6d440e1b13ba01896990675" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45683 } } }
I20260926 19:32:18.693244 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:18.693252 22181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:18.695739 22187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:18.700623 22187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:18.711297 22187 catalog_manager.cc:1347] Generated new cluster ID: 4884ea0ef95c42aeb92751ec43c765eb
I20260926 19:32:18.711490 22187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:18.725835 22187 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:18.726276 22187 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:18.732721 22187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04671e11e6d440e1b13ba01896990675: Generated new TSK 0
I20260926 19:32:18.733541 22187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:18.746160 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22103
W20260926 19:32:18.747256 22165 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48983
W20260926 19:32:18.748066 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22103: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:18.748212 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22103
2026-09-26T19:32:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:18 dist-test-slave-9wb0 krb5kdc[22201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:18 dist-test-slave-9wb0 krb5kdc[22201](info): set up 2 sockets
Sep 26 19:32:18 dist-test-slave-9wb0 krb5kdc[22201](info): commencing operation
krb5kdc: starting...
W20260926 19:32:19.877995 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:20.156766 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:20.492717 22214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:20.493101 22214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:20.493271 22214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:20.500769 22214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:20.500878 22214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:20.500989 22214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:20.501091 22214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:20.501160 22214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:20.511349 22214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:20.512794 22214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:20.514153 22214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:20.526233 22219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:20.526286 22220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:20.526311 22222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:20.527040 22214 server_base.cc:1034] running on GCE node
I20260926 19:32:20.527854 22214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:20.529349 22214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:20.530594 22214 hybrid_clock.cc:648] HybridClock initialized: now 1790451140530546 us; error 51 us; skew 500 ppm
I20260926 19:32:20.530948 22214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:20.534097 22214 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:20.535590 22214 webserver.cc:460] Webserver started at http://127.0.0.1:35601/ using document root <none> and password file <none>
I20260926 19:32:20.536415 22214 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:20.536577 22214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:20.536886 22214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:20.539918 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "da8bcb886a2e40c0b1d1a7220cca371e"
format_stamp: "Formatted at 2026-09-26 19:32:20 on dist-test-slave-9wb0"
server_key: "2b41908d7c76fb9b4dade19cd995d3bf"
server_key_iv: "8e90a97192e9a10430ecb652fb7f42a9"
server_key_version: "encryptionkey@0"
I20260926 19:32:20.540668 22214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "da8bcb886a2e40c0b1d1a7220cca371e"
format_stamp: "Formatted at 2026-09-26 19:32:20 on dist-test-slave-9wb0"
server_key: "2b41908d7c76fb9b4dade19cd995d3bf"
server_key_iv: "8e90a97192e9a10430ecb652fb7f42a9"
server_key_version: "encryptionkey@0"
I20260926 19:32:20.545671 22214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 19:32:20.548938 22229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:20.550320 22214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:20.550534 22214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "da8bcb886a2e40c0b1d1a7220cca371e"
format_stamp: "Formatted at 2026-09-26 19:32:20 on dist-test-slave-9wb0"
server_key: "2b41908d7c76fb9b4dade19cd995d3bf"
server_key_iv: "8e90a97192e9a10430ecb652fb7f42a9"
server_key_version: "encryptionkey@0"
I20260926 19:32:20.550734 22214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:20.582329 22214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:20.586464 22214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:20.586758 22214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:20.605520 22214 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45975
I20260926 19:32:20.605568 22280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45975 every 8 connection(s)
I20260926 19:32:20.606881 22214 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:20.610484 22281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:20.614099 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22214
I20260926 19:32:20.614289 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:20.614826 1180 external_mini_cluster.cc:1428] Setting key 016bbaa7565cd1b16787cbb6f3bff995
I20260926 19:32:20.623456 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e: Bootstrap starting.
I20260926 19:32:20.627370 22281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:20.628753 22281 log.cc:824] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e: Log is configured to *not* fsync() on all Append() calls
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:32:20.631963 22281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e: No bootstrap required, opened a new log
I20260926 19:32:20.639256 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:20.616630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:49459 (local address 127.0.0.1:45975)
0926 19:32:20.617430 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:20.617442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:20.617482 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:32:20.619113 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:20.619137 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:20.619553 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:20.619890 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:20.621780 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:20.623115 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:20.624897 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:20.625661 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:20.625792 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:20.631031 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:20.631038 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:20.631052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:20.631091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:20.634376 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:20.635325 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:20.635333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:20.635339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:20.635497 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:20.636008 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:20.636011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:20.636013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:20.636694 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:20.636995 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:20.637853 (+ 858us) server_negotiation.cc:299] Negotiation successful
0926 19:32:20.638176 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":368,"threads_started":1}
I20260926 19:32:20.641347 22281 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } }
I20260926 19:32:20.641784 22281 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:20.641909 22281 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8bcb886a2e40c0b1d1a7220cca371e, State: Initialized, Role: FOLLOWER
I20260926 19:32:20.642623 22281 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } }
I20260926 19:32:20.642905 22281 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:20.643069 22281 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:20.643245 22281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:20.643404 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:20.645535 22281 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } }
I20260926 19:32:20.646068 22281 leader_election.cc:302] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8bcb886a2e40c0b1d1a7220cca371e; no voters:
I20260926 19:32:20.646733 22281 leader_election.cc:288] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:20.646946 22289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:20.648438 22289 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [term 1 LEADER]: Becoming Leader. State: Replica: da8bcb886a2e40c0b1d1a7220cca371e, State: Running, Role: LEADER
I20260926 19:32:20.649101 22281 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:20.649091 22289 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } }
I20260926 19:32:20.652609 22292 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8bcb886a2e40c0b1d1a7220cca371e. Latest consensus state: current_term: 1 leader_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } } }
I20260926 19:32:20.652818 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8bcb886a2e40c0b1d1a7220cca371e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45975 } } }
I20260926 19:32:20.653100 22292 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:20.653204 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:20.654660 22297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:20.659986 22297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:20.676030 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:20.649457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:56499 (local address 127.0.0.1:45975)
0926 19:32:20.649658 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:20.649668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:20.653963 (+ 4295us) server_negotiation.cc:407] Connection header received
0926 19:32:20.654249 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:20.654255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:20.654342 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:20.654430 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:20.656694 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:20.658344 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:20.660812 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:20.661796 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:20.661920 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:20.666273 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:20.666283 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:20.666291 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:20.666347 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:20.671161 (+ 4814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:20.672421 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:20.672430 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:20.672436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:20.672663 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:20.673392 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:20.673400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:20.673406 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:20.674023 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:20.674356 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:20.675233 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 19:32:20.675518 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 19:32:20.681474 22297 catalog_manager.cc:1347] Generated new cluster ID: d9793a5f23bc4ca7abcae20a37592b7a
I20260926 19:32:20.681599 22297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:20.692569 22297 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:20.694406 22297 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:20.707618 22297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8bcb886a2e40c0b1d1a7220cca371e: Generated new TSK 0
I20260926 19:32:20.708521 22297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:20.722395 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22214
I20260926 19:32:20.724040 22276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:20.819265 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22214
W20260926 19:32:20.839668 22199 connection.cc:535] client connection to 127.0.0.1:45975 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45975): Connection reset by peer (error 104)
2026-09-26T19:32:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2073 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22313](info): set up 2 sockets
Sep 26 19:32:20 dist-test-slave-9wb0 krb5kdc[22313](info): commencing operation
krb5kdc: starting...
W20260926 19:32:21.941723 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:22 dist-test-slave-9wb0 krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:22.213091 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46011
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:22.547500 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:22.547791 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:22.547969 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:22.556063 22326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:22.556222 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:22.556342 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:22.556452 22326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:22.556547 22326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:22.566305 22326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:22.567615 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:22.568912 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:22.580698 22331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:22.580852 22332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:22.580906 22334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:22.581962 22326 server_base.cc:1034] running on GCE node
I20260926 19:32:22.582778 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:22.584487 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:22.585768 22326 hybrid_clock.cc:648] HybridClock initialized: now 1790451142585722 us; error 42 us; skew 500 ppm
I20260926 19:32:22.586153 22326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:22 dist-test-slave-9wb0 krb5kdc[22313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:22.589460 22326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:22.590828 22326 webserver.cc:460] Webserver started at http://127.0.0.1:34479/ using document root <none> and password file <none>
I20260926 19:32:22.591507 22326 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:22.591631 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:22.591938 22326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:22.594867 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "07eb4064a4a949cf848fc82454fbc49c"
format_stamp: "Formatted at 2026-09-26 19:32:22 on dist-test-slave-9wb0"
server_key: "a9a38d626033c0a18320622f4a4655df"
server_key_iv: "de6c3b34b096c0c3b364ac0650abe321"
server_key_version: "encryptionkey@0"
I20260926 19:32:22.595618 22326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "07eb4064a4a949cf848fc82454fbc49c"
format_stamp: "Formatted at 2026-09-26 19:32:22 on dist-test-slave-9wb0"
server_key: "a9a38d626033c0a18320622f4a4655df"
server_key_iv: "de6c3b34b096c0c3b364ac0650abe321"
server_key_version: "encryptionkey@0"
I20260926 19:32:22.600559 22326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 19:32:22.604051 22341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:22.605273 22326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:32:22.605552 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "07eb4064a4a949cf848fc82454fbc49c"
format_stamp: "Formatted at 2026-09-26 19:32:22 on dist-test-slave-9wb0"
server_key: "a9a38d626033c0a18320622f4a4655df"
server_key_iv: "de6c3b34b096c0c3b364ac0650abe321"
server_key_version: "encryptionkey@0"
I20260926 19:32:22.605734 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:22.638120 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:22.642529 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:22.642855 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:22.661708 22326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46011
I20260926 19:32:22.661746 22392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46011 every 8 connection(s)
I20260926 19:32:22.663030 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:22.666653 22393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:22.670204 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22326
I20260926 19:32:22.670392 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:22.670933 1180 external_mini_cluster.cc:1428] Setting key 8389a7484a19ea8ba90a4805606c7ff5
I20260926 19:32:22.679306 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c: Bootstrap starting.
I20260926 19:32:22.683112 22393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:22.684219 22393 log.cc:824] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:22.684327 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:22.672826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:42905 (local address 127.0.0.1:46011)
0926 19:32:22.673605 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:22.673617 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:22.673654 (+ 37us) server_negotiation.cc:407] Connection header received
0926 19:32:22.675255 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:22.675277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:22.675703 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:22.676034 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:22.676641 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:22.678073 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:22.679900 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:22.680543 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:22.680668 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:22.682107 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:22.682115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:22.682127 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:22.682640 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:22.682718 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:22.682742 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:22.682855 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:22.683157 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 19:32:22.683343 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":339,"threads_started":1}
I20260926 19:32:22.687201 22393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c: No bootstrap required, opened a new log
I20260926 19:32:22.688171 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:22.695751 22393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } }
I20260926 19:32:22.696201 22393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:22.696308 22393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07eb4064a4a949cf848fc82454fbc49c, State: Initialized, Role: FOLLOWER
I20260926 19:32:22.696985 22393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } }
I20260926 19:32:22.697274 22393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:22.697471 22393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:22.697652 22393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:22.699338 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:22.692492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:51597 (local address 127.0.0.1:46011)
0926 19:32:22.692737 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:22.692742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:22.693036 (+ 294us) server_negotiation.cc:407] Connection header received
0926 19:32:22.693191 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:22.693195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:22.693253 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:22.693465 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:22.694054 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:22.694825 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:22.696705 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:22.697167 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:22.697347 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:22.698546 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:22.698551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:22.698554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:22.698573 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:22.698629 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:22.698646 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:22.698735 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:22.698927 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 19:32:22.698969 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:32:22.700706 22393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } }
I20260926 19:32:22.701210 22393 leader_election.cc:302] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07eb4064a4a949cf848fc82454fbc49c; no voters:
I20260926 19:32:22.701831 22393 leader_election.cc:288] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:22.701982 22403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:22.702730 22403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [term 1 LEADER]: Becoming Leader. State: Replica: 07eb4064a4a949cf848fc82454fbc49c, State: Running, Role: LEADER
I20260926 19:32:22.703222 22403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } }
I20260926 19:32:22.703924 22393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:22.706211 22404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07eb4064a4a949cf848fc82454fbc49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } } }
I20260926 19:32:22.706642 22404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:22.707094 22405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07eb4064a4a949cf848fc82454fbc49c. Latest consensus state: current_term: 1 leader_uuid: "07eb4064a4a949cf848fc82454fbc49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07eb4064a4a949cf848fc82454fbc49c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46011 } } }
I20260926 19:32:22.707434 22405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:22.709980 22409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:22.715337 22409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:22.724597 22409 catalog_manager.cc:1347] Generated new cluster ID: 2fe13e5ca7ac461b998292038c3a9679
I20260926 19:32:22.724733 22409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:22.786068 22409 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:22.787324 22409 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:22.796597 22409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07eb4064a4a949cf848fc82454fbc49c: Generated new TSK 0
I20260926 19:32:22.797574 22409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:22.812003 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22326
W20260926 19:32:22.813192 22388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.105:42905
W20260926 19:32:22.814016 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:22.814170 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22326
W20260926 19:32:22.832302 22311 connection.cc:535] client connection to 127.0.0.1:46011 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46011): Connection reset by peer (error 104)
2026-09-26T19:32:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:22 dist-test-slave-9wb0 krb5kdc[22424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:22 dist-test-slave-9wb0 krb5kdc[22424](info): set up 2 sockets
Sep 26 19:32:22 dist-test-slave-9wb0 krb5kdc[22424](info): commencing operation
krb5kdc: starting...
W20260926 19:32:23.942032 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:24 dist-test-slave-9wb0 krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:24.224836 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:24.562479 22437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:24.562772 22437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:24.562933 22437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:24.570268 22437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:24.570375 22437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:24.570483 22437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:24.570575 22437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:24.570668 22437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:24.580773 22437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:24.582078 22437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:24.583364 22437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:24.594998 22445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:24.594998 22442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:24.594894 22443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:24.595669 22437 server_base.cc:1034] running on GCE node
I20260926 19:32:24.596477 22437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:24.597922 22437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:24.599160 22437 hybrid_clock.cc:648] HybridClock initialized: now 1790451144599111 us; error 36 us; skew 500 ppm
I20260926 19:32:24.599545 22437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:24 dist-test-slave-9wb0 krb5kdc[22424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:24.607882 22437 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:24.609527 22437 webserver.cc:460] Webserver started at http://127.0.0.1:36661/ using document root <none> and password file <none>
I20260926 19:32:24.610248 22437 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:24.610399 22437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:24.610729 22437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:24.613819 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "baf7540e893940ec8813497765a6f1f1"
format_stamp: "Formatted at 2026-09-26 19:32:24 on dist-test-slave-9wb0"
server_key: "6c62d6e8c4cd72be255e1f112762afbb"
server_key_iv: "5ff6a9437a16da43253ebb00e70dd8f9"
server_key_version: "encryptionkey@0"
I20260926 19:32:24.614586 22437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "baf7540e893940ec8813497765a6f1f1"
format_stamp: "Formatted at 2026-09-26 19:32:24 on dist-test-slave-9wb0"
server_key: "6c62d6e8c4cd72be255e1f112762afbb"
server_key_iv: "5ff6a9437a16da43253ebb00e70dd8f9"
server_key_version: "encryptionkey@0"
I20260926 19:32:24.619755 22437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260926 19:32:24.623296 22452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:24.624571 22437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:32:24.624750 22437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "baf7540e893940ec8813497765a6f1f1"
format_stamp: "Formatted at 2026-09-26 19:32:24 on dist-test-slave-9wb0"
server_key: "6c62d6e8c4cd72be255e1f112762afbb"
server_key_iv: "5ff6a9437a16da43253ebb00e70dd8f9"
server_key_version: "encryptionkey@0"
I20260926 19:32:24.624965 22437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:24.630776 22437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:24.634920 22437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:24.635219 22437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:24.654572 22437 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34961
I20260926 19:32:24.654611 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34961 every 8 connection(s)
I20260926 19:32:24.655977 22437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:24.659602 22504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:24.661806 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22437
I20260926 19:32:24.661983 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:24.662532 1180 external_mini_cluster.cc:1428] Setting key 4648fcc2eee758940f74353b0d488591
I20260926 19:32:24.672325 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:24.664703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:53995 (local address 127.0.0.1:34961)
0926 19:32:24.665735 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:24.665750 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:24.666101 (+ 351us) server_negotiation.cc:407] Connection header received
0926 19:32:24.668451 (+ 2350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:24.668484 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:24.668919 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:24.669398 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:24.670377 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:24.670385 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:24.670402 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:24.670435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:24.670500 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:24.670525 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:24.670757 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:24.670884 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 19:32:24.671098 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":446,"threads_started":1}
I20260926 19:32:24.673156 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1: Bootstrap starting.
I20260926 19:32:24.676054 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:24.676508 22504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:24.677802 22504 log.cc:824] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:24.681190 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1: No bootstrap required, opened a new log
I20260926 19:32:24.683949 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:24.681216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:57059 (local address 127.0.0.1:34961)
0926 19:32:24.681446 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:24.681455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:24.681922 (+ 467us) server_negotiation.cc:407] Connection header received
0926 19:32:24.682126 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:24.682130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:24.682182 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:24.682262 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:24.683266 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:24.683270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:24.683273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:24.683286 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:24.683324 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:24.683337 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:24.683446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:24.683565 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 19:32:24.683615 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 19:32:24.689589 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 19:32:24.690021 22504 raft_consensus.cc:374] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:24.690133 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf7540e893940ec8813497765a6f1f1, State: Initialized, Role: FOLLOWER
I20260926 19:32:24.690744 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 19:32:24.690992 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:24.691145 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:24.691311 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:24.693604 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 19:32:24.694132 22504 leader_election.cc:302] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: baf7540e893940ec8813497765a6f1f1; no voters:
I20260926 19:32:24.694772 22504 leader_election.cc:288] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:24.695073 22514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:24.696448 22514 raft_consensus.cc:686] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [term 1 LEADER]: Becoming Leader. State: Replica: baf7540e893940ec8813497765a6f1f1, State: Running, Role: LEADER
I20260926 19:32:24.697115 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:24.696971 22514 consensus_queue.cc:237] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260926 19:32:24.700057 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "baf7540e893940ec8813497765a6f1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260926 19:32:24.700084 22516 sys_catalog.cc:455] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader baf7540e893940ec8813497765a6f1f1. Latest consensus state: current_term: 1 leader_uuid: "baf7540e893940ec8813497765a6f1f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "baf7540e893940ec8813497765a6f1f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260926 19:32:24.700598 22516 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:24.700598 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:24.701683 22522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:24.709112 22522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:24.719707 22522 catalog_manager.cc:1347] Generated new cluster ID: 39a9f8429c95409b8d258dba76071811
I20260926 19:32:24.719839 22522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:24.732143 22522 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:24.732689 22522 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:24.747649 22522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P baf7540e893940ec8813497765a6f1f1: Generated new TSK 0
I20260926 19:32:24.748514 22522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:24.762970 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22437
W20260926 19:32:24.764325 22499 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.105:53995
W20260926 19:32:24.765054 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22437: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:24.765193 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22437
2026-09-26T19:32:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:24 dist-test-slave-9wb0 krb5kdc[22535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:24 dist-test-slave-9wb0 krb5kdc[22535](info): set up 2 sockets
Sep 26 19:32:24 dist-test-slave-9wb0 krb5kdc[22535](info): commencing operation
krb5kdc: starting...
W20260926 19:32:25.901894 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:26.189438 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:26.540729 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:26.541011 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:26.541105 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:26.548400 22548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:26.548516 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:26.548564 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:26.548619 22548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:26.548663 22548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:26.558580 22548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43319
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:26.559787 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:26.561035 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:26.572921 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:26.572962 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:26.572870 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:26.574004 22548 server_base.cc:1034] running on GCE node
I20260926 19:32:26.574745 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:26.576102 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:26.577376 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790451146577340 us; error 56 us; skew 500 ppm
I20260926 19:32:26.577745 22548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:26.580780 22548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:26.582381 22548 webserver.cc:460] Webserver started at http://127.0.0.1:38963/ using document root <none> and password file <none>
I20260926 19:32:26.583071 22548 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:26.583214 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:26.583520 22548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:26.586462 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "cddd0d57284c4822ae839bbd17081bd2"
format_stamp: "Formatted at 2026-09-26 19:32:26 on dist-test-slave-9wb0"
server_key: "06da15dafe96789956e816aab8e67f63"
server_key_iv: "bcf083cdf8a75157b2fefad60136caf9"
server_key_version: "encryptionkey@0"
I20260926 19:32:26.587217 22548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "cddd0d57284c4822ae839bbd17081bd2"
format_stamp: "Formatted at 2026-09-26 19:32:26 on dist-test-slave-9wb0"
server_key: "06da15dafe96789956e816aab8e67f63"
server_key_iv: "bcf083cdf8a75157b2fefad60136caf9"
server_key_version: "encryptionkey@0"
I20260926 19:32:26.592270 22548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:32:26.595733 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:26.597111 22548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:32:26.597326 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cddd0d57284c4822ae839bbd17081bd2"
format_stamp: "Formatted at 2026-09-26 19:32:26 on dist-test-slave-9wb0"
server_key: "06da15dafe96789956e816aab8e67f63"
server_key_iv: "bcf083cdf8a75157b2fefad60136caf9"
server_key_version: "encryptionkey@0"
I20260926 19:32:26.597543 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:26.613541 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:26.617952 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:26.618235 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:26.638324 22548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43319
I20260926 19:32:26.638356 22614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43319 every 8 connection(s)
I20260926 19:32:26.639752 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:26.643517 22615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:26.649451 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22548
I20260926 19:32:26.649652 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:26.650236 1180 external_mini_cluster.cc:1428] Setting key 2cf03ff0d4bc52b37cc23c8092cc5549
I20260926 19:32:26.655361 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2: Bootstrap starting.
I20260926 19:32:26.658576 22615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:26.659618 22615 log.cc:824] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:26.662462 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2: No bootstrap required, opened a new log
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 19:32:26.670548 22615 raft_consensus.cc:348] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260926 19:32:26.670953 22615 raft_consensus.cc:374] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:26.671092 22615 raft_consensus.cc:729] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddd0d57284c4822ae839bbd17081bd2, State: Initialized, Role: FOLLOWER
I20260926 19:32:26.671818 22615 consensus_queue.cc:260] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260926 19:32:26.672086 22615 raft_consensus.cc:388] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:26.672260 22615 raft_consensus.cc:482] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:26.672465 22615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:26.673033 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:26.652224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:48597 (local address 127.0.0.1:43319)
0926 19:32:26.653098 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:26.653111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:26.653154 (+ 43us) server_negotiation.cc:407] Connection header received
0926 19:32:26.654960 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:26.654986 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:26.655411 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:26.655863 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:26.657432 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:26.658756 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:26.660140 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:26.660853 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:26.661058 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:26.665122 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:26.665130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:26.665144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:26.665183 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:26.668407 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:26.669256 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:26.669262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:26.669269 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:26.669495 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:26.669966 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:26.669976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:26.669978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:26.670547 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:26.670889 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:26.671402 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 19:32:26.671790 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":406,"threads_started":1}
I20260926 19:32:26.675551 22615 raft_consensus.cc:504] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260926 19:32:26.676144 22615 leader_election.cc:302] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cddd0d57284c4822ae839bbd17081bd2; no voters:
I20260926 19:32:26.676844 22615 leader_election.cc:288] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:26.676999 22620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:26.677340 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:26.679116 22615 sys_catalog.cc:564] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:26.682920 22620 raft_consensus.cc:686] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [term 1 LEADER]: Becoming Leader. State: Replica: cddd0d57284c4822ae839bbd17081bd2, State: Running, Role: LEADER
I20260926 19:32:26.683640 22620 consensus_queue.cc:237] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } }
I20260926 19:32:26.687024 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cddd0d57284c4822ae839bbd17081bd2. Latest consensus state: current_term: 1 leader_uuid: "cddd0d57284c4822ae839bbd17081bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } } }
I20260926 19:32:26.687435 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:26.688010 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cddd0d57284c4822ae839bbd17081bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cddd0d57284c4822ae839bbd17081bd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43319 } } }
I20260926 19:32:26.688257 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:26.695400 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:26.700994 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:26.704744 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:26.686298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:57613 (local address 127.0.0.1:43319)
0926 19:32:26.686747 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:26.686755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:26.687361 (+ 606us) server_negotiation.cc:407] Connection header received
0926 19:32:26.687571 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:26.687580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:26.687708 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:26.687832 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:26.690441 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:26.692060 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:26.694663 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:26.695569 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:26.695733 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:26.698425 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:26.698430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:26.698434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:26.698466 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:26.701753 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:26.702597 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:26.702601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:26.702604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:26.702725 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:26.703224 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:26.703228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:26.703230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:26.703567 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:26.703880 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:26.704267 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 19:32:26.704453 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260926 19:32:26.711127 22637 catalog_manager.cc:1347] Generated new cluster ID: 89996471d3ed49aeb9f3dea48722697a
I20260926 19:32:26.711237 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:26.738806 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:26.740729 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:26.751169 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cddd0d57284c4822ae839bbd17081bd2: Generated new TSK 0
I20260926 19:32:26.752214 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:26.767489 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22548
I20260926 19:32:26.768787 22610 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:26.861357 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22548
W20260926 19:32:26.880030 22533 connection.cc:535] client connection to 127.0.0.1:43319 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43319): Connection reset by peer (error 104)
2026-09-26T19:32:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2096 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22647](info): set up 2 sockets
Sep 26 19:32:26 dist-test-slave-9wb0 krb5kdc[22647](info): commencing operation
krb5kdc: starting...
W20260926 19:32:27.973775 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:28 dist-test-slave-9wb0 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:28.258479 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43591
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:28.590477 22660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:28.590762 22660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:28.590915 22660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:28.598243 22660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:28.598349 22660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:28.598438 22660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:28.598546 22660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:28.598623 22660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:28.609616 22660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43591
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:28.610939 22660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:28.612219 22660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:28.623812 22666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:28.623849 22665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:28.623893 22668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:28.625051 22660 server_base.cc:1034] running on GCE node
I20260926 19:32:28.626137 22660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:28.627625 22660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:28.628866 22660 hybrid_clock.cc:648] HybridClock initialized: now 1790451148628825 us; error 52 us; skew 500 ppm
I20260926 19:32:28.629235 22660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:28 dist-test-slave-9wb0 krb5kdc[22647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:28.632429 22660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:28.633929 22660 webserver.cc:460] Webserver started at http://127.0.0.1:44399/ using document root <none> and password file <none>
I20260926 19:32:28.634579 22660 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:28.634682 22660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:28.634920 22660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:28.638391 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "7aace24903bd41b583d3e799611af6e3"
format_stamp: "Formatted at 2026-09-26 19:32:28 on dist-test-slave-9wb0"
server_key: "2249dbe4a8d62c3b6a093beb3f54ce2e"
server_key_iv: "aabd23575e5ef85708e67a7a223c412b"
server_key_version: "encryptionkey@0"
I20260926 19:32:28.639123 22660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "7aace24903bd41b583d3e799611af6e3"
format_stamp: "Formatted at 2026-09-26 19:32:28 on dist-test-slave-9wb0"
server_key: "2249dbe4a8d62c3b6a093beb3f54ce2e"
server_key_iv: "aabd23575e5ef85708e67a7a223c412b"
server_key_version: "encryptionkey@0"
I20260926 19:32:28.644209 22660 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:28.647722 22675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:28.649055 22660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 19:32:28.649230 22660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "7aace24903bd41b583d3e799611af6e3"
format_stamp: "Formatted at 2026-09-26 19:32:28 on dist-test-slave-9wb0"
server_key: "2249dbe4a8d62c3b6a093beb3f54ce2e"
server_key_iv: "aabd23575e5ef85708e67a7a223c412b"
server_key_version: "encryptionkey@0"
I20260926 19:32:28.649459 22660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:28.663990 22660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:28.668424 22660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:28.668733 22660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:28.688438 22660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43591
I20260926 19:32:28.688465 22726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43591 every 8 connection(s)
I20260926 19:32:28.689893 22660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:28.693863 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:28.695232 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22660
I20260926 19:32:28.695452 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:28.696125 1180 external_mini_cluster.cc:1428] Setting key 0863f1ce82fc0611402311c1157ee404
I20260926 19:32:28.707079 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3: Bootstrap starting.
I20260926 19:32:28.708947 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:28.698138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:56591 (local address 127.0.0.1:43591)
0926 19:32:28.698992 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:28.699004 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:28.699045 (+ 41us) server_negotiation.cc:407] Connection header received
0926 19:32:28.700659 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:28.700683 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:28.701056 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:28.701502 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:28.702001 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:28.703214 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:28.704571 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:28.705264 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:28.705513 (+ 249us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:28.706771 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:28.706778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:28.706792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:28.707353 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:28.707431 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:28.707459 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:28.707635 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:28.707886 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 19:32:28.708044 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":378,"threads_started":1}
I20260926 19:32:28.710793 22727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:28.712046 22727 log.cc:824] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:28.712759 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:28.714681 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3: No bootstrap required, opened a new log
I20260926 19:32:28.722482 22727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260926 19:32:28.722863 22727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:28.723004 22727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aace24903bd41b583d3e799611af6e3, State: Initialized, Role: FOLLOWER
I20260926 19:32:28.723258 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:28.717098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:38077 (local address 127.0.0.1:43591)
0926 19:32:28.717276 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:28.717283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:28.717652 (+ 369us) server_negotiation.cc:407] Connection header received
0926 19:32:28.717850 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:28.717854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:28.717913 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:28.718033 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:28.718510 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:28.719281 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:28.720664 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:28.721098 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:28.721239 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:28.722393 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:28.722397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:28.722401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:28.722417 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:28.722452 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:28.722469 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:28.722590 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:28.722780 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:32:28.722830 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:32:28.723692 22727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260926 19:32:28.723971 22727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:28.724143 22727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:28.724323 22727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:28.726641 22727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260926 19:32:28.727145 22727 leader_election.cc:302] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aace24903bd41b583d3e799611af6e3; no voters:
I20260926 19:32:28.727696 22727 leader_election.cc:288] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:28.727834 22737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:28.728543 22737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [term 1 LEADER]: Becoming Leader. State: Replica: 7aace24903bd41b583d3e799611af6e3, State: Running, Role: LEADER
I20260926 19:32:28.729176 22737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } }
I20260926 19:32:28.729892 22727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:28.732790 22738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aace24903bd41b583d3e799611af6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } } }
I20260926 19:32:28.733208 22738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:28.734798 22746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:28.733978 22739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aace24903bd41b583d3e799611af6e3. Latest consensus state: current_term: 1 leader_uuid: "7aace24903bd41b583d3e799611af6e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aace24903bd41b583d3e799611af6e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43591 } } }
I20260926 19:32:28.736503 22739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:28.739363 22746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:28.747641 22746 catalog_manager.cc:1347] Generated new cluster ID: 8ec5c6841b3f4bc99532ef434c5b3936
I20260926 19:32:28.747769 22746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:28.764483 22746 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:28.765839 22746 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:28.783753 22746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aace24903bd41b583d3e799611af6e3: Generated new TSK 0
I20260926 19:32:28.784857 22746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:28.794831 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22660
W20260926 19:32:28.796121 22722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.105:56591
W20260926 19:32:28.796873 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22660: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:28.797015 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22660
W20260926 19:32:28.815079 22645 connection.cc:535] client connection to 127.0.0.1:43591 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43591): Connection reset by peer (error 104)
2026-09-26T19:32:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:28 dist-test-slave-9wb0 krb5kdc[22758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:28 dist-test-slave-9wb0 krb5kdc[22758](info): set up 2 sockets
Sep 26 19:32:28 dist-test-slave-9wb0 krb5kdc[22758](info): commencing operation
krb5kdc: starting...
W20260926 19:32:29.941694 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:30 dist-test-slave-9wb0 krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 19:32:30.216974 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:30.552603 22771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:30.552872 22771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:30.553064 22771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:30.560233 22771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:30.560338 22771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:30.560449 22771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:30.560564 22771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:30.560648 22771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:30.570550 22771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:30.571828 22771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:30.573113 22771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:30.584916 22777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:30.584885 22779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:30.585047 22776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:30.586098 22771 server_base.cc:1034] running on GCE node
I20260926 19:32:30.586890 22771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:30.588318 22771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:30.589588 22771 hybrid_clock.cc:648] HybridClock initialized: now 1790451150589490 us; error 85 us; skew 500 ppm
I20260926 19:32:30.589962 22771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:30 dist-test-slave-9wb0 krb5kdc[22758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:30.599439 22771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 19:32:30.600975 22771 webserver.cc:460] Webserver started at http://127.0.0.1:44579/ using document root <none> and password file <none>
I20260926 19:32:30.601739 22771 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:30.601892 22771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:30.602203 22771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:30.605103 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "cf2e7188b79c45b1bd3284442c7047c0"
format_stamp: "Formatted at 2026-09-26 19:32:30 on dist-test-slave-9wb0"
server_key: "d13cdf6a3c1c1593fba17c18d98eb3e6"
server_key_iv: "c5aa33ca78e975a7a7c085d0ce67193a"
server_key_version: "encryptionkey@0"
I20260926 19:32:30.605916 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "cf2e7188b79c45b1bd3284442c7047c0"
format_stamp: "Formatted at 2026-09-26 19:32:30 on dist-test-slave-9wb0"
server_key: "d13cdf6a3c1c1593fba17c18d98eb3e6"
server_key_iv: "c5aa33ca78e975a7a7c085d0ce67193a"
server_key_version: "encryptionkey@0"
I20260926 19:32:30.610919 22771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:32:30.614262 22786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:30.615604 22771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:30.615783 22771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "cf2e7188b79c45b1bd3284442c7047c0"
format_stamp: "Formatted at 2026-09-26 19:32:30 on dist-test-slave-9wb0"
server_key: "d13cdf6a3c1c1593fba17c18d98eb3e6"
server_key_iv: "c5aa33ca78e975a7a7c085d0ce67193a"
server_key_version: "encryptionkey@0"
I20260926 19:32:30.615985 22771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:30.621769 22771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:30.625849 22771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:30.626158 22771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:30.644737 22771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35723
I20260926 19:32:30.644783 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35723 every 8 connection(s)
I20260926 19:32:30.646209 22771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:30.649750 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:30.653789 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22771
I20260926 19:32:30.653975 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:30.654520 1180 external_mini_cluster.cc:1428] Setting key fb16f54016363fb9d18b5632f3a499cc
I20260926 19:32:30.661013 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0: Bootstrap starting.
I20260926 19:32:30.663578 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:30.656359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:37997 (local address 127.0.0.1:35723)
0926 19:32:30.657178 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:30.657190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:30.657236 (+ 46us) server_negotiation.cc:407] Connection header received
0926 19:32:30.659751 (+ 2515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:30.659775 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:30.660120 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:30.660449 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:30.661554 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:30.661561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:30.661578 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:30.661595 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:30.661653 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:30.661672 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:30.661890 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:30.662042 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 19:32:30.662255 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":380,"threads_started":1}
I20260926 19:32:30.665040 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:30.666296 22838 log.cc:824] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:30.667701 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:30.669431 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0: No bootstrap required, opened a new log
I20260926 19:32:30.674630 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:30.672255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:39855 (local address 127.0.0.1:35723)
0926 19:32:30.672489 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:30.672495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:30.672733 (+ 238us) server_negotiation.cc:407] Connection header received
0926 19:32:30.672862 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:30.672872 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:30.672933 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:30.673059 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:30.673936 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:30.673942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:30.673947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:30.673966 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:30.674013 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:30.674034 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:30.674175 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:30.674228 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 19:32:30.674290 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:32:30.679083 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260926 19:32:30.679522 22838 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:30.679610 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf2e7188b79c45b1bd3284442c7047c0, State: Initialized, Role: FOLLOWER
I20260926 19:32:30.680135 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260926 19:32:30.680363 22838 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:30.680506 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:30.680693 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:30.683259 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260926 19:32:30.683725 22838 leader_election.cc:302] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf2e7188b79c45b1bd3284442c7047c0; no voters:
I20260926 19:32:30.684297 22838 leader_election.cc:288] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:30.684401 22848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:30.685190 22848 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [term 1 LEADER]: Becoming Leader. State: Replica: cf2e7188b79c45b1bd3284442c7047c0, State: Running, Role: LEADER
I20260926 19:32:30.685796 22848 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } }
I20260926 19:32:30.686826 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:30.689028 22849 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf2e7188b79c45b1bd3284442c7047c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } } }
I20260926 19:32:30.689459 22849 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:30.690229 22850 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf2e7188b79c45b1bd3284442c7047c0. Latest consensus state: current_term: 1 leader_uuid: "cf2e7188b79c45b1bd3284442c7047c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf2e7188b79c45b1bd3284442c7047c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35723 } } }
I20260926 19:32:30.690647 22850 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:30.694809 22854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:30.700924 22854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:30.710358 22854 catalog_manager.cc:1347] Generated new cluster ID: ac7a0c002be34ac6b9b075aeb505ebd8
I20260926 19:32:30.710489 22854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:30.732780 22854 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:30.733273 22854 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:30.744702 22854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf2e7188b79c45b1bd3284442c7047c0: Generated new TSK 0
I20260926 19:32:30.745611 22854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:30.751370 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22771
W20260926 19:32:30.753041 22833 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.105:37997
W20260926 19:32:30.753937 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22771: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:30.754133 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22771
2026-09-26T19:32:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:30 dist-test-slave-9wb0 krb5kdc[22869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:30 dist-test-slave-9wb0 krb5kdc[22869](info): set up 2 sockets
Sep 26 19:32:30 dist-test-slave-9wb0 krb5kdc[22869](info): commencing operation
krb5kdc: starting...
W20260926 19:32:31.870023 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:32.148947 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41887
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:41887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:32.481454 22882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:32.481745 22882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:32.481899 22882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:32.493062 22882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:32.493248 22882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:32.493355 22882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:32.493459 22882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:32.493533 22882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:32.503851 22882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:41887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:32.505142 22882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:32.506441 22882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:32.517874 22887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:32.517799 22888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:32.517838 22890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:32.518899 22882 server_base.cc:1034] running on GCE node
I20260926 19:32:32.519732 22882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:32.521077 22882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:32.522315 22882 hybrid_clock.cc:648] HybridClock initialized: now 1790451152522268 us; error 59 us; skew 500 ppm
I20260926 19:32:32.522665 22882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451152, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:32.525727 22882 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:32.527169 22882 webserver.cc:460] Webserver started at http://127.1.39.62:44647/ using document root <none> and password file <none>
I20260926 19:32:32.527822 22882 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:32.527966 22882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:32.528263 22882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:32.531195 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "b196d77d65544f848ea941e23b8611d2"
format_stamp: "Formatted at 2026-09-26 19:32:32 on dist-test-slave-9wb0"
server_key: "b525f16843f731c594b2d5bdad6f2759"
server_key_iv: "4b3077c9d3cef030287ed96f29e12e77"
server_key_version: "encryptionkey@0"
I20260926 19:32:32.531966 22882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "b196d77d65544f848ea941e23b8611d2"
format_stamp: "Formatted at 2026-09-26 19:32:32 on dist-test-slave-9wb0"
server_key: "b525f16843f731c594b2d5bdad6f2759"
server_key_iv: "4b3077c9d3cef030287ed96f29e12e77"
server_key_version: "encryptionkey@0"
I20260926 19:32:32.536927 22882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 19:32:32.540347 22897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:32.541685 22882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:32.541873 22882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "b196d77d65544f848ea941e23b8611d2"
format_stamp: "Formatted at 2026-09-26 19:32:32 on dist-test-slave-9wb0"
server_key: "b525f16843f731c594b2d5bdad6f2759"
server_key_iv: "4b3077c9d3cef030287ed96f29e12e77"
server_key_version: "encryptionkey@0"
I20260926 19:32:32.542073 22882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:32.570004 22882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:32.574162 22882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:32.574447 22882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:32.593214 22882 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:41887
I20260926 19:32:32.593266 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:41887 every 8 connection(s)
I20260926 19:32:32.594586 22882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:32.595009 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22882
I20260926 19:32:32.595187 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:32.595726 1180 external_mini_cluster.cc:1428] Setting key 9f0fdb4269dd1befbe98ff9787450d73
I20260926 19:32:32.599128 22949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:32.612246 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2: Bootstrap starting.
I20260926 19:32:32.616276 22949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:32.617595 22949 log.cc:824] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:32.620082 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:32.597657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44973 (local address 127.1.39.62:41887)
0926 19:32:32.598440 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:32.598451 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:32.598491 (+ 40us) server_negotiation.cc:407] Connection header received
0926 19:32:32.600161 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:32.600184 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:32.600554 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:32.600851 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:32.602630 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:32.603970 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:32.605861 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:32.606487 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:32.606645 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:32.611348 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:32.611358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:32.611378 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:32.611421 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:32.615303 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:32.616104 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:32.616115 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:32.616124 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:32.616341 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:32.616807 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:32.616812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:32.616815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:32.617533 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:32.617844 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:32.618440 (+ 596us) server_negotiation.cc:299] Negotiation successful
0926 19:32:32.618797 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":365,"threads_started":1}
I20260926 19:32:32.620960 22949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2: No bootstrap required, opened a new log
I20260926 19:32:32.624043 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:32.629413 22949 raft_consensus.cc:348] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } }
I20260926 19:32:32.629776 22949 raft_consensus.cc:374] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:32.629873 22949 raft_consensus.cc:729] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b196d77d65544f848ea941e23b8611d2, State: Initialized, Role: FOLLOWER
I20260926 19:32:32.630456 22949 consensus_queue.cc:260] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } }
I20260926 19:32:32.630687 22949 raft_consensus.cc:388] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:32.630837 22949 raft_consensus.cc:482] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:32.631019 22949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:32.633880 22949 raft_consensus.cc:504] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } }
I20260926 19:32:32.634472 22949 leader_election.cc:302] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b196d77d65544f848ea941e23b8611d2; no voters:
I20260926 19:32:32.635114 22949 leader_election.cc:288] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:32.635238 22959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:32.637166 22949 sys_catalog.cc:564] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:32.638183 22959 raft_consensus.cc:686] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [term 1 LEADER]: Becoming Leader. State: Replica: b196d77d65544f848ea941e23b8611d2, State: Running, Role: LEADER
I20260926 19:32:32.638780 22959 consensus_queue.cc:237] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } }
I20260926 19:32:32.642496 22961 sys_catalog.cc:455] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b196d77d65544f848ea941e23b8611d2. Latest consensus state: current_term: 1 leader_uuid: "b196d77d65544f848ea941e23b8611d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } } }
I20260926 19:32:32.642508 22960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b196d77d65544f848ea941e23b8611d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b196d77d65544f848ea941e23b8611d2" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41887 } } }
I20260926 19:32:32.643123 22960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:32.643148 22961 sys_catalog.cc:458] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:32.645503 22966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:32.646670 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:32.628270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56969 (local address 127.1.39.62:41887)
0926 19:32:32.628463 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:32.628471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:32.628883 (+ 412us) server_negotiation.cc:407] Connection header received
0926 19:32:32.629017 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:32.629040 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:32.629114 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:32.629219 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:32.631035 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:32.632242 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:32.633698 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:32.634331 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:32.634488 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:32.637949 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:32.637957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:32.637961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:32.637998 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:32.643152 (+ 5154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:32.644080 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:32.644087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:32.644092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:32.644371 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:32.644980 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:32.644985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:32.644989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:32.645398 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:32.645740 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:32.646120 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 19:32:32.646301 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:32:32.652065 22966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:32.660360 22966 catalog_manager.cc:1347] Generated new cluster ID: 45d6359b6d064a91a67c8a9672cc2e3d
I20260926 19:32:32.660494 22966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:32.676781 22966 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:32.678608 22966 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:32.690595 22966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b196d77d65544f848ea941e23b8611d2: Generated new TSK 0
I20260926 19:32:32.691497 22966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:32.702024 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22882
I20260926 19:32:32.703215 22944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:32.794646 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22882
W20260926 19:32:32.813012 22867 connection.cc:535] client connection to 127.1.39.62:41887 recv error: Network error: failed to read from TLS socket (remote: 127.1.39.62:41887): Connection reset by peer (error 104)
2026-09-26T19:32:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22981](info): set up 2 sockets
Sep 26 19:32:32 dist-test-slave-9wb0 krb5kdc[22981](info): commencing operation
krb5kdc: starting...
W20260926 19:32:33.909708 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:34 dist-test-slave-9wb0 krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:34.192341 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33701
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:33701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:34.528824 22994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:34.529184 22994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:34.529377 22994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:34.537284 22994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:34.537493 22994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:34.537585 22994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:34.537706 22994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:34.537788 22994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:34.547691 22994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:33701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:33701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:34.548985 22994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:34.550359 22994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:34.562435 22999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:34.562361 23002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:34.562389 23000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:34.563225 22994 server_base.cc:1034] running on GCE node
I20260926 19:32:34.563913 22994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:34.565218 22994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:34.566427 22994 hybrid_clock.cc:648] HybridClock initialized: now 1790451154566363 us; error 52 us; skew 500 ppm
I20260926 19:32:34.566778 22994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:34 dist-test-slave-9wb0 krb5kdc[22981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:34.569859 22994 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:34.571336 22994 webserver.cc:460] Webserver started at http://127.1.39.62:33079/ using document root <none> and password file <none>
I20260926 19:32:34.571996 22994 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:34.572146 22994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:34.572440 22994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:34.575418 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f"
format_stamp: "Formatted at 2026-09-26 19:32:34 on dist-test-slave-9wb0"
server_key: "12d1fd5bf9f04f561266e0087364d4ec"
server_key_iv: "1c35eee916ccc7650e6388208dd44585"
server_key_version: "encryptionkey@0"
I20260926 19:32:34.576181 22994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f"
format_stamp: "Formatted at 2026-09-26 19:32:34 on dist-test-slave-9wb0"
server_key: "12d1fd5bf9f04f561266e0087364d4ec"
server_key_iv: "1c35eee916ccc7650e6388208dd44585"
server_key_version: "encryptionkey@0"
I20260926 19:32:34.581229 22994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 19:32:34.584548 23009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:34.585819 22994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:34.585999 22994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f"
format_stamp: "Formatted at 2026-09-26 19:32:34 on dist-test-slave-9wb0"
server_key: "12d1fd5bf9f04f561266e0087364d4ec"
server_key_iv: "1c35eee916ccc7650e6388208dd44585"
server_key_version: "encryptionkey@0"
I20260926 19:32:34.586198 22994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:34.609740 22994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:34.613821 22994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:34.614112 22994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:34.632616 22994 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:33701
I20260926 19:32:34.632659 23060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:33701 every 8 connection(s)
I20260926 19:32:34.634080 22994 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:34.637873 23061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:34.639988 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 22994
I20260926 19:32:34.640164 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:34.640714 1180 external_mini_cluster.cc:1428] Setting key 38fbd771d3da657c384cca22594efec6
I20260926 19:32:34.652177 23061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f: Bootstrap starting.
I20260926 19:32:34.654320 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:34.642519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46587 (local address 127.1.39.62:33701)
0926 19:32:34.643365 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:34.643376 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:34.643418 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:32:34.645015 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:34.645040 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:34.645563 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:34.646040 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:34.646590 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:34.647846 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:34.649845 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:34.650532 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:34.650660 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:34.651985 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:34.651992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:34.652005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:34.652594 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:34.652674 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:34.652700 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:34.652847 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:34.653140 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 19:32:34.653375 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":410,"threads_started":1}
I20260926 19:32:34.655611 23061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:34.656697 23061 log.cc:824] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:34.657969 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:34.659432 23061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f: No bootstrap required, opened a new log
I20260926 19:32:34.669535 23064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:34.662304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46783 (local address 127.1.39.62:33701)
0926 19:32:34.662508 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:34.662514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:34.663077 (+ 563us) server_negotiation.cc:407] Connection header received
0926 19:32:34.663230 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:34.663233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:34.663293 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:34.663369 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:34.663950 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:34.664699 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:34.666568 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:34.667022 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:34.667116 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:34.668631 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:34.668635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:34.668639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:34.668672 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:34.668724 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:34.668740 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:34.668823 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:34.669047 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 19:32:34.669090 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:32:34.669207 23061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } }
I20260926 19:32:34.669727 23061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:34.669835 23061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50eb46f6ca5248a39e2c35d84c0a7a1f, State: Initialized, Role: FOLLOWER
I20260926 19:32:34.670531 23061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } }
I20260926 19:32:34.670807 23061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:34.670974 23061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:34.671157 23061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:34.673393 23061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } }
I20260926 19:32:34.673908 23061 leader_election.cc:302] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50eb46f6ca5248a39e2c35d84c0a7a1f; no voters:
I20260926 19:32:34.674582 23061 leader_election.cc:288] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:34.674808 23071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:34.675578 23071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [term 1 LEADER]: Becoming Leader. State: Replica: 50eb46f6ca5248a39e2c35d84c0a7a1f, State: Running, Role: LEADER
I20260926 19:32:34.676093 23071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } }
I20260926 19:32:34.676812 23061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:34.679422 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50eb46f6ca5248a39e2c35d84c0a7a1f. Latest consensus state: current_term: 1 leader_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } } }
I20260926 19:32:34.679405 23072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50eb46f6ca5248a39e2c35d84c0a7a1f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 33701 } } }
I20260926 19:32:34.679845 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:34.679859 23072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:34.681455 23080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:34.687530 23080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:34.697633 23080 catalog_manager.cc:1347] Generated new cluster ID: b3f42c8b333947c1a9d9a86370bc4f39
I20260926 19:32:34.697754 23080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:34.717960 23080 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:34.719657 23080 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:34.725492 23080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50eb46f6ca5248a39e2c35d84c0a7a1f: Generated new TSK 0
I20260926 19:32:34.726240 23080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:34.736531 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22994
W20260926 19:32:34.737636 23056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46587
W20260926 19:32:34.738328 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 22994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:34.738481 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 22994
W20260926 19:32:34.756495 22979 connection.cc:535] client connection to 127.1.39.62:33701 recv error: Network error: failed to read from TLS socket (remote: 127.1.39.62:33701): Connection reset by peer (error 104)
2026-09-26T19:32:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:34 dist-test-slave-9wb0 krb5kdc[23092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:34 dist-test-slave-9wb0 krb5kdc[23092](info): set up 2 sockets
Sep 26 19:32:34 dist-test-slave-9wb0 krb5kdc[23092](info): commencing operation
krb5kdc: starting...
W20260926 19:32:35.845810 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:36 dist-test-slave-9wb0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:36.122727 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:32877
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:32877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:36.451351 23105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:36.451647 23105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:36.451745 23105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:36.459079 23105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:36.459195 23105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:36.459244 23105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:36.459312 23105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:36.459349 23105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:36.469893 23105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:32877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:32877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:36.471164 23105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:36.472391 23105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:36.484597 23111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:36.484711 23113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:36.484735 23110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:36.485940 23105 server_base.cc:1034] running on GCE node
I20260926 19:32:36.487196 23105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:36.489535 23105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:36.490816 23105 hybrid_clock.cc:648] HybridClock initialized: now 1790451156490752 us; error 60 us; skew 500 ppm
I20260926 19:32:36.491287 23105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:36 dist-test-slave-9wb0 krb5kdc[23092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451156, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:36.499250 23105 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:36.501029 23105 webserver.cc:460] Webserver started at http://127.1.39.62:38411/ using document root <none> and password file <none>
I20260926 19:32:36.501878 23105 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:36.502038 23105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:36.502378 23105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:36.505743 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "6afc6c0d87ea487eb07da25850ccf6c7"
format_stamp: "Formatted at 2026-09-26 19:32:36 on dist-test-slave-9wb0"
server_key: "8b051b260546dbcbbaefa73a04d341b5"
server_key_iv: "c574c20ca1c83619cb1dd51ea380f6aa"
server_key_version: "encryptionkey@0"
I20260926 19:32:36.506620 23105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "6afc6c0d87ea487eb07da25850ccf6c7"
format_stamp: "Formatted at 2026-09-26 19:32:36 on dist-test-slave-9wb0"
server_key: "8b051b260546dbcbbaefa73a04d341b5"
server_key_iv: "c574c20ca1c83619cb1dd51ea380f6aa"
server_key_version: "encryptionkey@0"
I20260926 19:32:36.512446 23105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260926 19:32:36.515991 23120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:36.517387 23105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 19:32:36.517607 23105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "6afc6c0d87ea487eb07da25850ccf6c7"
format_stamp: "Formatted at 2026-09-26 19:32:36 on dist-test-slave-9wb0"
server_key: "8b051b260546dbcbbaefa73a04d341b5"
server_key_iv: "c574c20ca1c83619cb1dd51ea380f6aa"
server_key_version: "encryptionkey@0"
I20260926 19:32:36.517812 23105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:36.523505 23105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:36.527504 23105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:36.527808 23105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:36.546208 23105 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:32877
I20260926 19:32:36.546258 23171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:32877 every 8 connection(s)
I20260926 19:32:36.547580 23105 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:36.548575 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23105
I20260926 19:32:36.548792 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:36.549326 1180 external_mini_cluster.cc:1428] Setting key a12f310c2f6cf1e190c58d102ef96b9f
I20260926 19:32:36.551968 23172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:36.556687 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:36.551079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60919 (local address 127.1.39.62:32877)
0926 19:32:36.551876 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:36.551887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:36.551926 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:32:36.553597 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:36.553619 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:36.553913 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:36.554236 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:36.555251 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:36.555256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:36.555267 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:36.555280 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:36.555349 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:36.555367 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:36.555545 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:36.555645 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 19:32:36.555816 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":357,"threads_started":1}
I20260926 19:32:36.560067 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:36.564422 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7: Bootstrap starting.
I20260926 19:32:36.567876 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:36.565401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32931 (local address 127.1.39.62:32877)
0926 19:32:36.565539 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:36.565543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:36.565948 (+ 405us) server_negotiation.cc:407] Connection header received
0926 19:32:36.566177 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:36.566184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:36.566232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:36.566307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:36.567230 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:36.567234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:36.567237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:36.567250 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:36.567287 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:36.567301 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:36.567387 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:36.567521 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 19:32:36.567563 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 19:32:36.568450 23172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:36.569530 23172 log.cc:824] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:36.572314 23172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7: No bootstrap required, opened a new log
I20260926 19:32:36.579506 23172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } }
I20260926 19:32:36.579872 23172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:36.580010 23172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afc6c0d87ea487eb07da25850ccf6c7, State: Initialized, Role: FOLLOWER
I20260926 19:32:36.580626 23172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } }
I20260926 19:32:36.580855 23172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:36.580991 23172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:36.581163 23172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:36.583238 23172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } }
I20260926 19:32:36.583732 23172 leader_election.cc:302] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6afc6c0d87ea487eb07da25850ccf6c7; no voters:
I20260926 19:32:36.584251 23172 leader_election.cc:288] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:36.584448 23182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:36.585083 23182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [term 1 LEADER]: Becoming Leader. State: Replica: 6afc6c0d87ea487eb07da25850ccf6c7, State: Running, Role: LEADER
I20260926 19:32:36.585595 23182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } }
I20260926 19:32:36.586189 23172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:36.588259 23183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } } }
I20260926 19:32:36.588284 23184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6afc6c0d87ea487eb07da25850ccf6c7. Latest consensus state: current_term: 1 leader_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afc6c0d87ea487eb07da25850ccf6c7" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 32877 } } }
I20260926 19:32:36.588755 23183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:36.588755 23184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:36.591033 23187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:36.596242 23187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:36.604844 23187 catalog_manager.cc:1347] Generated new cluster ID: 1a69555ad03642498da65051fd65c548
I20260926 19:32:36.604960 23187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:36.657598 23187 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:36.658054 23187 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:36.668883 23187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6afc6c0d87ea487eb07da25850ccf6c7: Generated new TSK 0
I20260926 19:32:36.669921 23187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:36.687060 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23105
W20260926 19:32:36.688194 23167 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60919
W20260926 19:32:36.688911 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23105: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:36.689069 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23105
2026-09-26T19:32:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1950 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:36 dist-test-slave-9wb0 krb5kdc[23203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:36 dist-test-slave-9wb0 krb5kdc[23203](info): set up 2 sockets
Sep 26 19:32:36 dist-test-slave-9wb0 krb5kdc[23203](info): commencing operation
krb5kdc: starting...
W20260926 19:32:37.797832 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:37 dist-test-slave-9wb0 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:38.080626 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37593
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:42269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:37593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:38.412372 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:38.412659 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:38.412832 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:38.420089 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:38.420204 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:38.420320 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:38.420404 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:38.420485 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:38.430258 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:42269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:37593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:37593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:38.431543 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:38.432878 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:38.444703 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:38.444766 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:38.444796 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:38.445875 23216 server_base.cc:1034] running on GCE node
I20260926 19:32:38.446774 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:38.448269 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:38.449538 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790451158449444 us; error 102 us; skew 500 ppm
I20260926 19:32:38.449963 23216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:38 dist-test-slave-9wb0 krb5kdc[23203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451158, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:38.453609 23216 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:38.455025 23216 webserver.cc:460] Webserver started at http://127.1.39.62:35651/ using document root <none> and password file <none>
I20260926 19:32:38.455703 23216 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:38.455850 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:38.456180 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:38.459168 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9"
format_stamp: "Formatted at 2026-09-26 19:32:38 on dist-test-slave-9wb0"
server_key: "18926b40e057ed63f4ab0017e4a0dd70"
server_key_iv: "cba5cc79e7193152a35a8c4cafc9f692"
server_key_version: "encryptionkey@0"
I20260926 19:32:38.459932 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9"
format_stamp: "Formatted at 2026-09-26 19:32:38 on dist-test-slave-9wb0"
server_key: "18926b40e057ed63f4ab0017e4a0dd70"
server_key_iv: "cba5cc79e7193152a35a8c4cafc9f692"
server_key_version: "encryptionkey@0"
I20260926 19:32:38.465833 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 19:32:38.469177 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:38.470573 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:38.470749 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9"
format_stamp: "Formatted at 2026-09-26 19:32:38 on dist-test-slave-9wb0"
server_key: "18926b40e057ed63f4ab0017e4a0dd70"
server_key_iv: "cba5cc79e7193152a35a8c4cafc9f692"
server_key_version: "encryptionkey@0"
I20260926 19:32:38.470943 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:38.493968 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:38.498344 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:38.498677 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:38.516952 23216 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:37593
I20260926 19:32:38.516971 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:37593 every 8 connection(s)
I20260926 19:32:38.518354 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:38.522024 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:38.527349 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23216
I20260926 19:32:38.527523 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:38.527968 1180 external_mini_cluster.cc:1428] Setting key 32b8416aca7dc749de812a3dce8af75a
I20260926 19:32:38.533792 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9: Bootstrap starting.
I20260926 19:32:38.536831 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:38.537897 23283 log.cc:824] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:38.540629 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9: No bootstrap required, opened a new log
Sep 26 19:32:38 dist-test-slave-9wb0 krb5kdc[23203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:38.550295 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:38.529917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59229 (local address 127.1.39.62:37593)
0926 19:32:38.530910 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:38.530922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:38.530978 (+ 56us) server_negotiation.cc:407] Connection header received
0926 19:32:38.532708 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:38.532729 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:38.533100 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:38.533520 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:38.534951 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:38.536236 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:38.537639 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:38.538447 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:38.538654 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:38.543139 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:38.543147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:38.543160 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:38.543198 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:38.546428 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:38.547257 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:38.547266 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:38.547273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:38.547450 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:38.547813 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:38.547816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:38.547817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:38.548308 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:38.548526 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:38.548944 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 19:32:38.549284 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":779,"thread_start_us":411,"threads_started":1}
I20260926 19:32:38.550135 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } }
I20260926 19:32:38.550503 23283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:38.550676 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7bf0d2a47844498f52f67d7a7b8ea9, State: Initialized, Role: FOLLOWER
I20260926 19:32:38.551225 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } }
I20260926 19:32:38.551468 23283 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:38.551623 23283 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:38.551801 23283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:38.554087 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:38.554526 23283 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } }
I20260926 19:32:38.554988 23283 leader_election.cc:302] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a7bf0d2a47844498f52f67d7a7b8ea9; no voters:
I20260926 19:32:38.555580 23283 leader_election.cc:288] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:38.555701 23288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:38.557751 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:38.560417 23288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [term 1 LEADER]: Becoming Leader. State: Replica: 7a7bf0d2a47844498f52f67d7a7b8ea9, State: Running, Role: LEADER
I20260926 19:32:38.560997 23288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } }
I20260926 19:32:38.564043 23290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7bf0d2a47844498f52f67d7a7b8ea9. Latest consensus state: current_term: 1 leader_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } } }
I20260926 19:32:38.564405 23290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:38.564144 23289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7bf0d2a47844498f52f67d7a7b8ea9" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 37593 } } }
I20260926 19:32:38.564541 23289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:38.565635 23301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:38.575959 23301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:38.578912 23286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:38.559912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35147 (local address 127.1.39.62:37593)
0926 19:32:38.560104 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:38.560113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:38.560139 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:32:38.560377 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:38.560390 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:38.560503 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:38.560590 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:38.563175 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:38.564527 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:38.566483 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:38.567257 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:38.567410 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:38.570973 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:38.570981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:38.570987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:38.571022 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:38.575056 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:38.575942 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:38.575948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:38.575952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:38.576107 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:38.576547 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:38.576552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:38.576555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:38.576936 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:38.577197 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:38.578305 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0926 19:32:38.578542 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 19:32:38.586222 23301 catalog_manager.cc:1347] Generated new cluster ID: f1b0a6199dd64045ab2d32cbd65b5da5
I20260926 19:32:38.586349 23301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:38.602980 23301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:38.604887 23301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:38.616433 23301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a7bf0d2a47844498f52f67d7a7b8ea9: Generated new TSK 0
I20260926 19:32:38.617277 23301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:38.626494 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23216
I20260926 19:32:38.628072 23278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:38.719374 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23216
2026-09-26T19:32:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2027 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:38 dist-test-slave-9wb0 krb5kdc[23315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:38 dist-test-slave-9wb0 krb5kdc[23315](info): set up 2 sockets
Sep 26 19:32:38 dist-test-slave-9wb0 krb5kdc[23315](info): commencing operation
krb5kdc: starting...
W20260926 19:32:39.845762 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:40 dist-test-slave-9wb0 krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:40.130208 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41801
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:41801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:40.461735 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:40.462034 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:40.462222 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:40.469762 23328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:40.469871 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:40.469976 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:40.470100 23328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:40.470207 23328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:40.480338 23328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:41801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:40.481783 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:40.483067 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:40.498590 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:40.498613 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:40.499310 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:40.499382 23328 server_base.cc:1034] running on GCE node
I20260926 19:32:40.500272 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:40.501789 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:40.503032 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790451160502998 us; error 46 us; skew 500 ppm
I20260926 19:32:40.503409 23328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:40 dist-test-slave-9wb0 krb5kdc[23315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451160, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:40.506588 23328 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:40.508154 23328 webserver.cc:460] Webserver started at http://127.1.39.62:44233/ using document root <none> and password file <none>
I20260926 19:32:40.508847 23328 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:40.508997 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:40.509366 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:40.512317 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "ef18100e723d46528fb48b3e943eebbe"
format_stamp: "Formatted at 2026-09-26 19:32:40 on dist-test-slave-9wb0"
server_key: "b207794226e2f042834970dac97fae33"
server_key_iv: "859c2a6806ed4a96ecedebbac9186974"
server_key_version: "encryptionkey@0"
I20260926 19:32:40.513098 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "ef18100e723d46528fb48b3e943eebbe"
format_stamp: "Formatted at 2026-09-26 19:32:40 on dist-test-slave-9wb0"
server_key: "b207794226e2f042834970dac97fae33"
server_key_iv: "859c2a6806ed4a96ecedebbac9186974"
server_key_version: "encryptionkey@0"
I20260926 19:32:40.518366 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 19:32:40.521872 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:40.523146 23328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260926 19:32:40.523335 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "ef18100e723d46528fb48b3e943eebbe"
format_stamp: "Formatted at 2026-09-26 19:32:40 on dist-test-slave-9wb0"
server_key: "b207794226e2f042834970dac97fae33"
server_key_iv: "859c2a6806ed4a96ecedebbac9186974"
server_key_version: "encryptionkey@0"
I20260926 19:32:40.523543 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:40.545194 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:40.549438 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:40.549741 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:40.568933 23328 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:41801
I20260926 19:32:40.568965 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:41801 every 8 connection(s)
I20260926 19:32:40.570372 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:40.574106 23395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:40.577370 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23328
I20260926 19:32:40.577560 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:40.577987 1180 external_mini_cluster.cc:1428] Setting key 982d53680cc8da68a9635af0e3558419
I20260926 19:32:40.585544 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe: Bootstrap starting.
I20260926 19:32:40.589107 23395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:40.590268 23395 log.cc:824] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:40.591310 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:40.579821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38555 (local address 127.1.39.62:41801)
0926 19:32:40.580543 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:40.580554 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:40.580593 (+ 39us) server_negotiation.cc:407] Connection header received
0926 19:32:40.582298 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:40.582318 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:40.582733 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:40.583104 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:40.583653 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:40.584908 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:40.586388 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:40.587520 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:40.587718 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:40.588867 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:40.588874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:40.588887 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:40.589540 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:40.589631 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:40.589657 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:40.589780 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:40.590005 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 19:32:40.590292 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":325,"threads_started":1}
I20260926 19:32:40.592947 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe: No bootstrap required, opened a new log
I20260926 19:32:40.594843 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:40.603026 23395 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } }
I20260926 19:32:40.603471 23395 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:40.603567 23395 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef18100e723d46528fb48b3e943eebbe, State: Initialized, Role: FOLLOWER
I20260926 19:32:40.604190 23395 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } }
I20260926 19:32:40.604437 23395 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:40.604606 23395 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:40.604787 23395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:40.606650 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:40.599918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50081 (local address 127.1.39.62:41801)
0926 19:32:40.600165 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:40.600176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:40.600576 (+ 400us) server_negotiation.cc:407] Connection header received
0926 19:32:40.600756 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:40.600760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:40.600826 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:40.600913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:40.601541 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:40.602245 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:40.604044 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:40.604614 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:40.604754 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:40.605899 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:40.605903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:40.605907 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:40.605926 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:40.605961 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:40.605976 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:40.606054 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:40.606210 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 19:32:40.606279 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:32:40.607817 23395 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } }
I20260926 19:32:40.608302 23395 leader_election.cc:302] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef18100e723d46528fb48b3e943eebbe; no voters:
I20260926 19:32:40.608857 23395 leader_election.cc:288] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:40.608976 23405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:40.609699 23405 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [term 1 LEADER]: Becoming Leader. State: Replica: ef18100e723d46528fb48b3e943eebbe, State: Running, Role: LEADER
I20260926 19:32:40.610280 23405 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } }
I20260926 19:32:40.610935 23395 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:40.613173 23406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef18100e723d46528fb48b3e943eebbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } } }
I20260926 19:32:40.613656 23406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:40.613651 23407 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef18100e723d46528fb48b3e943eebbe. Latest consensus state: current_term: 1 leader_uuid: "ef18100e723d46528fb48b3e943eebbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef18100e723d46528fb48b3e943eebbe" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 41801 } } }
I20260926 19:32:40.615607 23407 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:40.616304 23414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:40.621716 23414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:40.630709 23414 catalog_manager.cc:1347] Generated new cluster ID: 821d1c5642764f3c890b1acbfca32697
I20260926 19:32:40.630821 23414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:40.665859 23414 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:40.667342 23414 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:40.674531 23414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef18100e723d46528fb48b3e943eebbe: Generated new TSK 0
I20260926 19:32:40.675338 23414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:40.687652 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23328
W20260926 19:32:40.688735 23390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38555
W20260926 19:32:40.689465 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:40.689601 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23328
2026-09-26T19:32:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1972 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:40 dist-test-slave-9wb0 krb5kdc[23426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:40 dist-test-slave-9wb0 krb5kdc[23426](info): set up 2 sockets
Sep 26 19:32:40 dist-test-slave-9wb0 krb5kdc[23426](info): commencing operation
krb5kdc: starting...
W20260926 19:32:41.814275 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:41 dist-test-slave-9wb0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:42.102782 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38633
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:37689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:42.440071 23439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:42.440368 23439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:42.440459 23439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:42.447602 23439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:42.447715 23439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:42.447760 23439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:42.447818 23439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:42.447862 23439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:42.457901 23439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:37689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:42.459172 23439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:42.460527 23439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:42.473690 23444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:42.473755 23445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:42.473963 23447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:42.474748 23439 server_base.cc:1034] running on GCE node
I20260926 19:32:42.475469 23439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:42.476847 23439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:42.478045 23439 hybrid_clock.cc:648] HybridClock initialized: now 1790451162478009 us; error 36 us; skew 500 ppm
I20260926 19:32:42.478353 23439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:42 dist-test-slave-9wb0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451162, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:42.488981 23439 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:42.491546 23439 webserver.cc:460] Webserver started at http://127.1.39.62:38363/ using document root <none> and password file <none>
I20260926 19:32:42.492277 23439 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:42.492432 23439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:42.492750 23439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:42.495822 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "f86b0be5317648f3a0a6a92f2517188f"
format_stamp: "Formatted at 2026-09-26 19:32:42 on dist-test-slave-9wb0"
server_key: "6ad64df89e1cbfd8553817a205ceb51f"
server_key_iv: "bfc237fe58c94c3b44cc5f3ae8f36a7f"
server_key_version: "encryptionkey@0"
I20260926 19:32:42.496585 23439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "f86b0be5317648f3a0a6a92f2517188f"
format_stamp: "Formatted at 2026-09-26 19:32:42 on dist-test-slave-9wb0"
server_key: "6ad64df89e1cbfd8553817a205ceb51f"
server_key_iv: "bfc237fe58c94c3b44cc5f3ae8f36a7f"
server_key_version: "encryptionkey@0"
I20260926 19:32:42.502076 23439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:32:42.505452 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:42.506678 23439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 19:32:42.506855 23439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "f86b0be5317648f3a0a6a92f2517188f"
format_stamp: "Formatted at 2026-09-26 19:32:42 on dist-test-slave-9wb0"
server_key: "6ad64df89e1cbfd8553817a205ceb51f"
server_key_iv: "bfc237fe58c94c3b44cc5f3ae8f36a7f"
server_key_version: "encryptionkey@0"
I20260926 19:32:42.507102 23439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:42.512730 23439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:42.516571 23439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:42.516867 23439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:42.535290 23439 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:38633
I20260926 19:32:42.535322 23505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:38633 every 8 connection(s)
I20260926 19:32:42.536736 23439 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:42.540556 23506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:42.541357 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23439
I20260926 19:32:42.541536 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:42.541990 1180 external_mini_cluster.cc:1428] Setting key 40fc67d2b43695f27f123d882fe49f35
I20260926 19:32:42.551198 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:42.543780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36185 (local address 127.1.39.62:38633)
0926 19:32:42.544617 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:42.544634 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:42.544685 (+ 51us) server_negotiation.cc:407] Connection header received
0926 19:32:42.547404 (+ 2719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:42.547437 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:42.547904 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:42.548370 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:42.549324 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:42.549332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:42.549349 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:42.549365 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:42.549430 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:42.549460 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:42.549666 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:42.549795 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 19:32:42.549976 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":330,"threads_started":1}
I20260926 19:32:42.555220 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:42.555945 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f: Bootstrap starting.
I20260926 19:32:42.559989 23506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:42.561352 23506 log.cc:824] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:42.563091 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:42.560334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42169 (local address 127.1.39.62:38633)
0926 19:32:42.560540 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:42.560550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:42.561047 (+ 497us) server_negotiation.cc:407] Connection header received
0926 19:32:42.561156 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:42.561161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:42.561219 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:42.561377 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:42.562366 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:42.562372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:42.562376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 19:32:42.562393 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:42.562444 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 19:32:42.562463 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:42.562601 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:42.562736 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 19:32:42.562791 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 19:32:42.564630 23506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f: No bootstrap required, opened a new log
I20260926 19:32:42.571987 23506 raft_consensus.cc:348] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } }
I20260926 19:32:42.572384 23506 raft_consensus.cc:374] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 19:32:42.572501 23506 raft_consensus.cc:729] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f86b0be5317648f3a0a6a92f2517188f, State: Initialized, Role: FOLLOWER
I20260926 19:32:42.573048 23506 consensus_queue.cc:260] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } }
I20260926 19:32:42.573330 23506 raft_consensus.cc:388] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 19:32:42.573501 23506 raft_consensus.cc:482] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 19:32:42.573668 23506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:42.576009 23506 raft_consensus.cc:504] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } }
I20260926 19:32:42.576502 23506 leader_election.cc:302] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f86b0be5317648f3a0a6a92f2517188f; no voters:
I20260926 19:32:42.577095 23506 leader_election.cc:288] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 19:32:42.577245 23516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:42.578403 23516 raft_consensus.cc:686] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [term 1 LEADER]: Becoming Leader. State: Replica: f86b0be5317648f3a0a6a92f2517188f, State: Running, Role: LEADER
I20260926 19:32:42.579015 23516 consensus_queue.cc:237] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } }
I20260926 19:32:42.579335 23506 sys_catalog.cc:564] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:42.582244 23517 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f86b0be5317648f3a0a6a92f2517188f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } } }
I20260926 19:32:42.582070 23518 sys_catalog.cc:455] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f86b0be5317648f3a0a6a92f2517188f. Latest consensus state: current_term: 1 leader_uuid: "f86b0be5317648f3a0a6a92f2517188f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f86b0be5317648f3a0a6a92f2517188f" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38633 } } }
I20260926 19:32:42.582751 23517 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:42.582751 23518 sys_catalog.cc:458] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:42.585794 23523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:42.591219 23523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:42.600844 23523 catalog_manager.cc:1347] Generated new cluster ID: 424c458b4cc64e50be6a03d68dd8cf07
I20260926 19:32:42.600957 23523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:42.615758 23523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:42.616175 23523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:42.625612 23523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f86b0be5317648f3a0a6a92f2517188f: Generated new TSK 0
I20260926 19:32:42.626470 23523 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:42.633445 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23439
W20260926 19:32:42.634687 23501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36185
W20260926 19:32:42.635423 1180 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23439: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 19:32:42.635572 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23439
2026-09-26T19:32:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1945 ms)
[----------] 18 tests from AuthTokenIssuingTest (35823 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:42 dist-test-slave-9wb0 krb5kdc[23537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:42 dist-test-slave-9wb0 krb5kdc[23537](info): set up 2 sockets
Sep 26 19:32:42 dist-test-slave-9wb0 krb5kdc[23537](info): commencing operation
krb5kdc: starting...
W20260926 19:32:43.766345 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:43 dist-test-slave-9wb0 krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:44.055100 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42219
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:44.384655 23550 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:44.385088 23550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:44.385245 23550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:44.385408 23550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:44.392789 23550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:44.392933 23550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:44.393049 23550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:44.393150 23550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:44.393252 23550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:44.403245 23550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:42219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:44.404637 23550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:44.406029 23550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:44.418282 23556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:44.418362 23558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:44.418381 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:44.419163 23550 server_base.cc:1034] running on GCE node
I20260926 19:32:44.419915 23550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:44.421499 23550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:44.422760 23550 hybrid_clock.cc:648] HybridClock initialized: now 1790451164422693 us; error 29 us; skew 500 ppm
I20260926 19:32:44.423107 23550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:44 dist-test-slave-9wb0 krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:44.426277 23550 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:44.427697 23550 webserver.cc:460] Webserver started at http://127.1.39.62:35359/ using document root <none> and password file <none>
I20260926 19:32:44.428375 23550 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:44.428501 23550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:44.428817 23550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:44.431778 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "9f51db77609b498db77278597aaa4980"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "6218486d376a554bba5f81e1dce91563"
server_key_iv: "5a0c8c4dcab791f0d9d97e278b595553"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.432559 23550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "9f51db77609b498db77278597aaa4980"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "6218486d376a554bba5f81e1dce91563"
server_key_iv: "5a0c8c4dcab791f0d9d97e278b595553"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.437788 23550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:32:44.441160 23565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:44.442544 23550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:44.442744 23550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "9f51db77609b498db77278597aaa4980"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "6218486d376a554bba5f81e1dce91563"
server_key_iv: "5a0c8c4dcab791f0d9d97e278b595553"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.442961 23550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:44.466905 23550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:44.471421 23550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:44.471704 23550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:44.495795 23550 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:42219
I20260926 19:32:44.495815 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:42219 every 8 connection(s)
I20260926 19:32:44.497123 23550 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:44.500756 23617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:44.501883 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23550
I20260926 19:32:44.502106 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:44.502760 1180 external_mini_cluster.cc:1428] Setting key 483262471d407f619075abcbf6c33f49
I20260926 19:32:44.507504 23617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:44 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.39.61@KRBTEST.COM; defaulting to no policy
I20260926 19:32:44.532697 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.510761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.1.39.62:42219)
0926 19:32:44.512649 (+ 1888us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:44.512656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:44.512694 (+ 38us) server_negotiation.cc:407] Connection header received
0926 19:32:44.514365 (+ 1671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:44.514375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:44.514679 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:44.514880 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:44.516028 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:44.518043 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:44.519440 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:44.520439 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:44.520638 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:44.524236 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:44.524248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:44.524267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:44.524309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:44.528451 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:44.529175 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:44.529184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:44.529188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:44.529410 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:44.529808 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:44.529815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:44.529819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:44.530559 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:44.530780 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:44.531332 (+ 552us) server_negotiation.cc:299] Negotiation successful
0926 19:32:44.531732 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":1701,"spinlock_wait_cycles":16128,"thread_start_us":1466,"threads_started":1}
I20260926 19:32:44.532721 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.510761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42219 (local address 127.0.0.1:33974)
0926 19:32:44.511733 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.511784 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:44.514006 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:44.514932 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:44.514938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:44.515485 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:44.515813 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:44.515829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:44.518195 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:44.518199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:44.519294 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:44.519307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:44.520686 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:44.520691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:44.520813 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:44.521565 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:44.521588 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:44.523986 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:44.528621 (+ 4635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:44.528632 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:44.528642 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:44.529034 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:44.529524 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:44.529528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:44.529530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:44.529699 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:44.530713 (+ 1014us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:44.530719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:44.530909 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 19:32:44.531138 (+ 229us) client_negotiation.cc:240] Negotiation successful
0926 19:32:44.531517 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":419,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.61@KRBTEST.COM" created.
I20260926 19:32:44.536067 23617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:44.537468 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.536885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37922)
0926 19:32:44.537101 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.537269 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 19:32:44.537783 23566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
I20260926 19:32:44.539505 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.539027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37924)
0926 19:32:44.539218 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.539326 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 19:32:44.539765 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44881: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
I20260926 19:32:44.568425 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:44881
--webserver_interface=127.1.39.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:32:44.589119 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } attempt: 1
I20260926 19:32:44.590270 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.589837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37930)
0926 19:32:44.590024 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.590117 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:32:44.591230 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.590897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37942)
0926 19:32:44.591032 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.591100 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:32:44.591410 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44881: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
I20260926 19:32:44.659663 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } attempt: 2
I20260926 19:32:44.660804 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.660323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37956)
0926 19:32:44.660578 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.660663 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 19:32:44.661792 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.661444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37970)
0926 19:32:44.661585 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.661665 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:32:44.661972 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44881: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
I20260926 19:32:44.753213 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } attempt: 3
I20260926 19:32:44.754319 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.753886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37980)
0926 19:32:44.754097 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.754184 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:32:44.755287 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.754933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:37992)
0926 19:32:44.755096 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.755159 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:32:44.755463 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44881: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
W20260926 19:32:44.891162 23624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:44.891542 23624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:44.891670 23624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:44.891804 23624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 19:32:44.898694 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } attempt: 4
W20260926 19:32:44.899006 23624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:44.899127 23624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:44.899230 23624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:44.899322 23624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:44.899413 23624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:44.900004 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.899364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:38000)
0926 19:32:44.899739 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.899846 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260926 19:32:44.901002 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:44.900619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:38012)
0926 19:32:44.900769 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:44.900862 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:32:44.901185 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:44881: Network error: Client connection negotiation failed: client connection to 127.1.39.61:44881: connect: Connection refused (error 111)
I20260926 19:32:44.909912 23624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:44881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.39.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:44.911123 23624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:44.912322 23624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:44.923420 23630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:44.923550 23629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:44.923396 23632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:44.924156 23624 server_base.cc:1034] running on GCE node
I20260926 19:32:44.924839 23624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:44.926162 23624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:44.927371 23624 hybrid_clock.cc:648] HybridClock initialized: now 1790451164927312 us; error 32 us; skew 500 ppm
I20260926 19:32:44.927717 23624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:44 dist-test-slave-9wb0 krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:44.931129 23624 init.cc:377] Logged in from keytab as kudu/127.1.39.61@KRBTEST.COM (short username kudu)
I20260926 19:32:44.932668 23624 webserver.cc:460] Webserver started at http://127.1.39.61:40091/ using document root <none> and password file <none>
I20260926 19:32:44.933434 23624 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:44.933540 23624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:44.933864 23624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:44.936803 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data/instance:
uuid: "d1a8825ef619461cbd08aa8ada97ba3a"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "ab680eca96bb1a6d3cae108a5645c5f6"
server_key_iv: "11e517fd84d262a8b80ca443ebca2026"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.937633 23624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal/instance:
uuid: "d1a8825ef619461cbd08aa8ada97ba3a"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "ab680eca96bb1a6d3cae108a5645c5f6"
server_key_iv: "11e517fd84d262a8b80ca443ebca2026"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.942569 23624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 19:32:44.945890 23639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:44.947311 23624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:44.947492 23624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "d1a8825ef619461cbd08aa8ada97ba3a"
format_stamp: "Formatted at 2026-09-26 19:32:44 on dist-test-slave-9wb0"
server_key: "ab680eca96bb1a6d3cae108a5645c5f6"
server_key_iv: "11e517fd84d262a8b80ca443ebca2026"
server_key_version: "encryptionkey@0"
I20260926 19:32:44.947719 23624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:44.968948 23624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:44.972726 23624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:44.972981 23624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:44.990077 23624 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.61:44881
I20260926 19:32:44.990108 23690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.61:44881 every 8 connection(s)
I20260926 19:32:44.991400 23624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
I20260926 19:32:44.994256 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23624
I20260926 19:32:44.994436 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-1/wal/instance
I20260926 19:32:44.994844 1180 external_mini_cluster.cc:1428] Setting key 814224e0bc91304716843aa07c6fefdc
I20260926 19:32:44.995060 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:45.001277 23691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.39.60@KRBTEST.COM; defaulting to no policy
I20260926 19:32:45.022560 23620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.003896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33978 (local address 127.1.39.62:42219)
0926 19:32:45.004076 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.004080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.005373 (+ 1293us) server_negotiation.cc:407] Connection header received
0926 19:32:45.007171 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.007180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.007286 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.007411 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.008785 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.009984 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.011429 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.012217 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.012353 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.016290 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.016294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.016298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.016330 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.019123 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.020058 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.020061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.020064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.020173 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.020650 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.020653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.020655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.020908 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.021026 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.021990 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.022162 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 19:32:45.023387 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.004367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42219 (local address 127.0.0.1:33978)
0926 19:32:45.005167 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.005217 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.006900 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.007595 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.007602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.008080 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.008611 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.008627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.010129 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.010133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.011291 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.011302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.012416 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.012420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.012633 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.013323 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.013357 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.016052 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.019268 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.019278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.019296 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.019946 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.020281 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.020286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.020290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.020537 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.021045 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.021053 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.021596 (+ 543us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.021875 (+ 279us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.022296 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":358,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.60@KRBTEST.COM" created.
I20260926 19:32:45.025393 23691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:45.045102 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.026182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:38016)
0926 19:32:45.026343 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.026361 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.026445 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.030196 (+ 3751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.030204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.030734 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.030952 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.030967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.032361 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.032368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.033904 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.033920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.034758 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.034769 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.034985 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.035610 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.035637 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.038244 (+ 2607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.042732 (+ 4488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.042736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.042739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.043097 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.043529 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.043532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.043535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.043666 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.044431 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.044434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.044553 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.044657 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.044855 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 19:32:45.045650 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.026271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.1.39.61:44881)
0926 19:32:45.029773 (+ 3502us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.029779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.029804 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:32:45.029846 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.029852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.030027 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.030153 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.031153 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.032198 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.034043 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.034586 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.034686 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.038479 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.038496 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.038518 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.038562 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.042591 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.043233 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.043245 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.043251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.043415 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.043768 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.043776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.043782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.044282 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.044510 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.044855 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.045162 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3350,"thread_start_us":3185,"threads_started":1}
I20260926 19:32:45.048095 23691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:45.049356 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.048870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60510)
0926 19:32:45.049052 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.049173 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 19:32:45.049633 23642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.051276 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.050917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60520)
0926 19:32:45.051087 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.051178 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 19:32:45.051584 23691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
I20260926 19:32:45.063832 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:34005
--webserver_interface=127.1.39.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:32:45.100978 23691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 1
I20260926 19:32:45.102118 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.101695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60530)
0926 19:32:45.101880 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.101968 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 19:32:45.103022 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.102704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60540)
0926 19:32:45.102848 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.102922 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:32:45.103227 23691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.171451 23691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 2
I20260926 19:32:45.172525 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.172097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60548)
0926 19:32:45.172288 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.172380 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 19:32:45.173449 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.173091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60560)
0926 19:32:45.173219 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.173331 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:32:45.173631 23691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.200453 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } attempt: 5
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:45.215363 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.201382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.1.39.61:44881)
0926 19:32:45.201557 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.201561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.201580 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:32:45.201767 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.201771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.201849 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.201930 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.202961 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.203681 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.205459 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.205923 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.206027 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.209975 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.209980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.209983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.210016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.212657 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.213558 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.213563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.213565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.213701 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.214097 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.214100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.214103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.214390 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.214511 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.214931 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.215130 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:32:45.215473 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.201228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:38024)
0926 19:32:45.201487 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.201510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.201641 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.201997 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.202002 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.202548 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.202803 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.202816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.203841 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.203849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.205258 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.205273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.206077 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.206082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.206297 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.206953 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.206982 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.209784 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.212800 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.212807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.212811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.213394 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.213818 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.213823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.213827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.213986 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.214516 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.214521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.214686 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.214831 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.215114 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 19:32:45.216631 23617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:45.217734 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.217354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60564)
0926 19:32:45.217504 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.217582 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 19:32:45.218675 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.218334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60576)
0926 19:32:45.218491 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.218559 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 19:32:45.218973 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.264875 23691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 3
I20260926 19:32:45.266127 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.265632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60578)
0926 19:32:45.265857 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.265964 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 19:32:45.267218 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.266765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60586)
0926 19:32:45.266985 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.267070 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260926 19:32:45.267494 23691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.270215 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 1
I20260926 19:32:45.271279 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.270833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60594)
0926 19:32:45.271016 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.271115 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 19:32:45.272325 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.272023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60600)
0926 19:32:45.272161 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.272226 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 19:32:45.272620 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.340843 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 2
I20260926 19:32:45.342057 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.341595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60616)
0926 19:32:45.341787 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.341887 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 19:32:45.343080 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.342757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60624)
0926 19:32:45.342891 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.342977 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 19:32:45.343348 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
W20260926 19:32:45.402086 23698 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:45.402433 23698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:45.402561 23698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:45.402690 23698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:45.410000 23698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:45.410080 23698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:45.410176 23698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:45.410250 23698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:45.410333 23698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:45.410709 23691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 4
I20260926 19:32:45.411839 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.411358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60640)
0926 19:32:45.411610 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.411695 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 19:32:45.412742 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.412436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60652)
0926 19:32:45.412581 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.412644 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 19:32:45.412940 23691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.420204 23698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:42219,127.1.39.61:44881,127.1.39.60:34005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:34005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.39.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:45.421485 23698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:45.422748 23698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:45.433738 23703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:45.434501 23704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:45.435191 23706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:45.436209 23698 server_base.cc:1034] running on GCE node
I20260926 19:32:45.436983 23698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:45.438423 23698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:45.439694 23698 hybrid_clock.cc:648] HybridClock initialized: now 1790451165439618 us; error 49 us; skew 500 ppm
I20260926 19:32:45.440043 23698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:45.443125 23698 init.cc:377] Logged in from keytab as kudu/127.1.39.60@KRBTEST.COM (short username kudu)
I20260926 19:32:45.444460 23698 webserver.cc:460] Webserver started at http://127.1.39.60:35621/ using document root <none> and password file <none>
I20260926 19:32:45.445112 23698 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:45.445223 23698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:45.445580 23698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:45.448608 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data/instance:
uuid: "02c1b9f194b74c7dbd0f51f3e93691d7"
format_stamp: "Formatted at 2026-09-26 19:32:45 on dist-test-slave-9wb0"
server_key: "27f0bc497ec06fc640864b15ca019e36"
server_key_iv: "b0465ff93ddbcb34e36f1bed0a153732"
server_key_version: "encryptionkey@0"
I20260926 19:32:45.449438 23698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal/instance:
uuid: "02c1b9f194b74c7dbd0f51f3e93691d7"
format_stamp: "Formatted at 2026-09-26 19:32:45 on dist-test-slave-9wb0"
server_key: "27f0bc497ec06fc640864b15ca019e36"
server_key_iv: "b0465ff93ddbcb34e36f1bed0a153732"
server_key_version: "encryptionkey@0"
I20260926 19:32:45.449584 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 3
I20260926 19:32:45.450656 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.450095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60664)
0926 19:32:45.450335 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.450436 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:32:45.451813 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.451351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60680)
0926 19:32:45.451567 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.451657 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260926 19:32:45.452039 23617 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:34005: Network error: Client connection negotiation failed: client connection to 127.1.39.60:34005: connect: Connection refused (error 111)
I20260926 19:32:45.455233 23698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 19:32:45.458499 23713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:45.459787 23698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:45.459956 23698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "02c1b9f194b74c7dbd0f51f3e93691d7"
format_stamp: "Formatted at 2026-09-26 19:32:45 on dist-test-slave-9wb0"
server_key: "27f0bc497ec06fc640864b15ca019e36"
server_key_iv: "b0465ff93ddbcb34e36f1bed0a153732"
server_key_version: "encryptionkey@0"
I20260926 19:32:45.460170 23698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:45.497352 23698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:45.501560 23698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:45.501827 23698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:45.519438 23698 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.60:34005
I20260926 19:32:45.519460 23764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.60:34005 every 8 connection(s)
I20260926 19:32:45.520740 23698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
I20260926 19:32:45.521739 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23698
I20260926 19:32:45.521919 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790450801920901-1180-0/minicluster-data/master-2/wal/instance
I20260926 19:32:45.522348 1180 external_mini_cluster.cc:1428] Setting key 0dda966354ea45ec6aac613fe02bb41c
I20260926 19:32:45.525023 23765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:45.533720 23765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:45.546182 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.524209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.1.39.60:34005)
0926 19:32:45.525029 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.525045 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.525098 (+ 53us) server_negotiation.cc:407] Connection header received
0926 19:32:45.527452 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.527479 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.527957 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.528381 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.529733 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.531784 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.533110 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.534161 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.534349 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.539064 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.539073 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.539083 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.539115 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.542024 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.542831 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.542837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.542843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.543008 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.543406 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.543409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.543411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.543952 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.544246 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.544551 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.544967 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":311,"threads_started":1}
I20260926 19:32:45.551445 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.536369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:42219 (local address 127.0.0.1:33988)
0926 19:32:45.537199 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.537236 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.537422 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.537797 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.537803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.538666 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.539012 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.539027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.540064 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.540069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.541165 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.541176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.541972 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.541978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.542198 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.542774 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.542801 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.545284 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.548694 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.548704 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.548715 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.549281 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.549731 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.549734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.549736 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.549852 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.550392 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.550399 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.550557 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.550720 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.551007 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":497,"threads_started":1}
I20260926 19:32:45.551523 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.536377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.1.39.62:42219)
0926 19:32:45.536919 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.536927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.537341 (+ 414us) server_negotiation.cc:407] Connection header received
0926 19:32:45.537558 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.537562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.537626 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.537738 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.539164 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.539907 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.541355 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.541777 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.541902 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.545470 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.545474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.545478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.545508 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.548571 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.549419 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.549424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.549427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.549618 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.549987 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.549990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.549992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.550263 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.550418 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.550835 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.551032 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260926 19:32:45.552868 23765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:45.565178 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.548961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.1.39.62:42219)
0926 19:32:45.549616 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.549621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.549644 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:32:45.549689 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.549695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.549759 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.549866 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.551166 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.552407 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.553787 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.554376 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.554464 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.559042 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.559049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.559054 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.559093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.562357 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.563032 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.563037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.563040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.563177 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.563504 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.563511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.563518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.564031 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.564309 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.564613 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.564767 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":254,"threads_started":1}
I20260926 19:32:45.567169 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.553570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:44881 (local address 127.0.0.1:38026)
0926 19:32:45.553714 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.553733 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.553819 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.554075 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.554077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.554378 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.554540 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.554550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.555467 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.555473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.556949 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.556965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.557736 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.557741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.557859 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.558371 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.558387 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.560456 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.564283 (+ 3827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.564289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.564293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.564783 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.565186 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.565191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.565195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.565717 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.566303 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.566306 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.566429 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.566536 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.566833 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 19:32:45.567337 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.553597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.1.39.61:44881)
0926 19:32:45.553788 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.553794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.553812 (+ 18us) server_negotiation.cc:407] Connection header received
0926 19:32:45.553902 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.553907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.553974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.554077 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.554679 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.555319 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.557101 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.557599 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.557736 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.560682 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.560691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.560695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.560739 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.564150 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.564903 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.564909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.564912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.565057 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.565831 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.565839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.565843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.566170 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.566334 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.566659 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.566874 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 19:32:45.568939 23765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451165, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:45.582033 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.569642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60702)
0926 19:32:45.569800 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.569819 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.569912 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.570196 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.570200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.570508 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.570637 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.570644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.571628 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.571633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.572729 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.572740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.573524 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.573530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.573706 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.574174 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.574188 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.576197 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.579664 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.579668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.579671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.580008 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.580388 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.580395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.580401 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.580554 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.581137 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.581145 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.581316 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.581474 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.581700 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 19:32:45.582347 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.569781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.1.39.60:34005)
0926 19:32:45.569925 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.569930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.569945 (+ 15us) server_negotiation.cc:407] Connection header received
0926 19:32:45.569996 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.570001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.570081 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.570180 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.570762 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.571487 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.572879 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.573375 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.573510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.576357 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.576361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.576364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.576389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.579525 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.580117 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.580123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.580128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.580272 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.580674 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.580677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.580679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.580986 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.581151 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.581597 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.581898 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":46}
I20260926 19:32:45.582897 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.566914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.1.39.61:44881)
0926 19:32:45.567644 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.567668 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.567690 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:32:45.567776 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.567784 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.567915 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.568030 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.569921 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.570844 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.572339 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.572721 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.572808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.577029 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.577034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.577037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.577102 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.580191 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.580910 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.580915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.580918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.581064 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.581434 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.581437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.581454 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.581782 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.582050 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.582377 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.582598 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":1382,"threads_started":1}
I20260926 19:32:45.587837 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: Bootstrap starting.
I20260926 19:32:45.587939 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:45.591486 23765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:45.592489 23765 log.cc:824] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:45.591204 23592 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9f51db77609b498db77278597aaa4980"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34002
I20260926 19:32:45.595150 23765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: No bootstrap required, opened a new log
I20260926 19:32:45.601917 23765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.602226 23765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c1b9f194b74c7dbd0f51f3e93691d7, State: Initialized, Role: FOLLOWER
I20260926 19:32:45.602624 23765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.603426 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.603729 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.604450 23765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 19:32:45.613202 23785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:32:45.613355 23785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:32:45.611063 23698 master_runner.cc:186] Error getting master registration for 127.1.39.62:42219: OK, instance_id { permanent_uuid: "9f51db77609b498db77278597aaa4980" instance_seqno: 1790451164488117 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:32:45.622274 23698 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:32:45.629257 23617 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 4
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:32:45.644285 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.630385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60718 (local address 127.1.39.60:34005)
0926 19:32:45.630593 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.630597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.630611 (+ 14us) server_negotiation.cc:407] Connection header received
0926 19:32:45.630661 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.630664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.630717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.630823 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.631585 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.632311 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.633582 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.634056 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.634191 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.637701 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.637709 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.637714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.637751 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.641330 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.642119 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.642124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.642128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.642279 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.642701 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.642706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.642709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.643129 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.643291 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.643672 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.643900 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:32:45.644289 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.630002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60718)
0926 19:32:45.630252 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.630274 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.630400 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.630857 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.630862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.631256 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.631438 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.631447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.632445 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.632453 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.633444 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.633456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.634205 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.634211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.634427 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.635020 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.635045 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.637520 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.641474 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.641480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.641484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.642004 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.642392 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.642396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.642400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.642588 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.643253 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.643258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.643418 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.643569 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.643868 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 19:32:45.649132 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: Bootstrap starting.
I20260926 19:32:45.652509 23617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:45.653558 23617 log.cc:824] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:45.656323 23617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: No bootstrap required, opened a new log
I20260926 19:32:45.663728 23617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.664121 23617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f51db77609b498db77278597aaa4980, State: Initialized, Role: FOLLOWER
I20260926 19:32:45.664618 23617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.665843 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.666505 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.667081 23617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:45.667124 23592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 19:32:45.667315 23592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:45.671375 23592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.673735 23592 leader_election.cc:288] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [CANDIDATE]: Term 1 election: Requested vote from peers d1a8825ef619461cbd08aa8ada97ba3a (127.1.39.61:44881), 02c1b9f194b74c7dbd0f51f3e93691d7 (127.1.39.60:34005)
I20260926 19:32:45.674391 23740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f51db77609b498db77278597aaa4980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7"
I20260926 19:32:45.674906 23740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:45.678648 23740 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f51db77609b498db77278597aaa4980 in term 1.
I20260926 19:32:45.679452 23568 leader_election.cc:302] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02c1b9f194b74c7dbd0f51f3e93691d7, 9f51db77609b498db77278597aaa4980; no voters:
I20260926 19:32:45.680433 23788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:45.688421 23788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [term 1 LEADER]: Becoming Leader. State: Replica: 9f51db77609b498db77278597aaa4980, State: Running, Role: LEADER
I20260926 19:32:45.689159 23788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.698930 23800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51db77609b498db77278597aaa4980. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.699499 23800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:45.701776 23805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:45.712110 23691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } attempt: 5
I20260926 19:32:45.719820 23805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 19:32:45.725397 23571 tablet.cc:2369] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:45.730260 23811 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Sep 26 19:32:45 dist-test-slave-9wb0 krb5kdc[23537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451164, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:32:45.741521 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.713518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.1.39.60:34005)
0926 19:32:45.722062 (+ 8544us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.722069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.722095 (+ 26us) server_negotiation.cc:407] Connection header received
0926 19:32:45.722160 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.722166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.722244 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.722384 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.725240 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.726471 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.728117 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.728798 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.728935 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.733649 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.733656 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.733661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.733703 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.737367 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.738612 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.738619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.738622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.738796 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.739478 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.739484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.739488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.739872 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.740012 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.740817 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.741067 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8407,"thread_start_us":8261,"threads_started":1}
I20260926 19:32:45.742967 23694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.712728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:34005 (local address 127.0.0.1:60730)
0926 19:32:45.713143 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:45.713164 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:45.713272 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:45.722430 (+ 9158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:45.722437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:45.722951 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:45.723163 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.723176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.726606 (+ 3430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.726612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.727996 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:45.728009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.728992 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.729000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:45.729226 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.729979 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:45.730004 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:45.733340 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:45.737610 (+ 4270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.737615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.737619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.738416 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.739001 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:45.739006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:45.739009 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:45.739259 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:45.740151 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:45.740158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:45.740414 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 19:32:45.741910 (+ 1496us) client_negotiation.cc:240] Negotiation successful
0926 19:32:45.742377 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292}
I20260926 19:32:45.749172 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: Bootstrap starting.
I20260926 19:32:45.752317 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.703752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.1.39.61:44881)
0926 19:32:45.703954 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.703963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.705369 (+ 1406us) server_negotiation.cc:407] Connection header received
0926 19:32:45.706536 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.706545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.706656 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.706745 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.724636 (+ 17891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.725724 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.729402 (+ 3678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.730079 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.730180 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.738737 (+ 8557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.738743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.738748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.738781 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.742366 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.745397 (+ 3031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.745403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.745406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.745572 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.748085 (+ 2513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.748090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.748093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.748433 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.748722 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.751544 (+ 2822us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.751869 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 19:32:45.753842 23719 tablet.cc:2369] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:45.758622 23691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:45.761073 23691 log.cc:824] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:45.738472 23550 master_runner.cc:186] Error getting master registration for 127.1.39.61:44881: OK, instance_id { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" instance_seqno: 1790451164985022 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:32:45.764066 23550 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:32:45.771214 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.704234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60722 (local address 127.1.39.60:34005)
0926 19:32:45.704436 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.704445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.709196 (+ 4751us) server_negotiation.cc:407] Connection header received
0926 19:32:45.709681 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.709686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.709782 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.709861 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.711950 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.713155 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.723813 (+ 10658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.724583 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.724718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.754646 (+ 29928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.754655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.754666 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.754717 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.759017 (+ 4300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.766338 (+ 7321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.766345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.766351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.766561 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.767200 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.767209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.767216 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.767724 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.768043 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.770510 (+ 2467us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.770798 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 19:32:45.772454 23740 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [term 1 FOLLOWER]: Refusing update from remote peer 9f51db77609b498db77278597aaa4980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:45.773247 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:45.705559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.1.39.62:42219)
0926 19:32:45.705729 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:45.705736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:45.724142 (+ 18406us) server_negotiation.cc:407] Connection header received
0926 19:32:45.727781 (+ 3639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:45.727788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:45.727911 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:45.727984 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:45.734117 (+ 6133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.735343 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.743804 (+ 8461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:45.744529 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:45.744636 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:45.760053 (+ 15417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:45.760060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:45.760065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:45.760114 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:45.765078 (+ 4964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.768699 (+ 3621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.768710 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.768717 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.768909 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:45.769495 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:45.769504 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:45.769511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:45.769946 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:45.770223 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:45.772632 (+ 2409us) server_negotiation.cc:299] Negotiation successful
0926 19:32:45.772892 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 19:32:45.775877 23788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:45.793253 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: No bootstrap required, opened a new log
I20260926 19:32:45.793362 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51db77609b498db77278597aaa4980. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.793685 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.806010 23800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.806320 23800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:45.806906 23814 mvcc.cc:205] Tried to move back new op lower bound from 7333687974989889536 to 7333687974684454912. Current Snapshot: MvccSnapshot[applied={T|T < 7333687974989889536}]
I20260926 19:32:45.808358 23805 catalog_manager.cc:1347] Generated new cluster ID: 17ca343dc66f43728ceb7d38654c1c0c
I20260926 19:32:45.808461 23805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:45.808812 23774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.809057 23774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.812104 23691 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.812534 23691 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1a8825ef619461cbd08aa8ada97ba3a, State: Initialized, Role: FOLLOWER
I20260926 19:32:45.813021 23691 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } }
I20260926 19:32:45.818789 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.819640 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.820070 23691 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:45.820433 23665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:45.824208 23666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9f51db77609b498db77278597aaa4980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1a8825ef619461cbd08aa8ada97ba3a"
I20260926 19:32:45.831146 23665 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [term 1 FOLLOWER]: Refusing update from remote peer 9f51db77609b498db77278597aaa4980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:45.835912 23666 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f51db77609b498db77278597aaa4980 in term 1.
I20260926 19:32:45.837152 23788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:45.870071 23805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:45.871992 23805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:45.884128 23805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980: Generated new TSK 0
I20260926 19:32:45.885255 23805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:45.909683 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51db77609b498db77278597aaa4980. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.910072 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.910555 23827 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.911022 23827 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:45.912813 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f51db77609b498db77278597aaa4980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f51db77609b498db77278597aaa4980" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 42219 } } peers { permanent_uuid: "d1a8825ef619461cbd08aa8ada97ba3a" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 44881 } } peers { permanent_uuid: "02c1b9f194b74c7dbd0f51f3e93691d7" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 34005 } } }
I20260926 19:32:45.913201 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51db77609b498db77278597aaa4980 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:45.968006 23835 catalog_manager.cc:1259] Loaded cluster ID: 17ca343dc66f43728ceb7d38654c1c0c
I20260926 19:32:45.968221 23835 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: loading cluster ID for follower catalog manager: success
I20260926 19:32:45.972711 23835 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: acquiring CA information for follower catalog manager: success
I20260926 19:32:45.975608 23835 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1a8825ef619461cbd08aa8ada97ba3a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 19:32:46.006497 1180 mini_kdc.cc:348] Time spent kdestroy: real 0.104s user 0.000s sys 0.031s
W20260926 19:32:46.062299 23695 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.046224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38048 (local address 127.1.39.61:44881)
0926 19:32:46.046444 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.046451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.060170 (+ 13719us) server_negotiation.cc:407] Connection header received
0926 19:32:46.060559 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.060567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.060660 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.060751 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.062022 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38048: BlockingRecv error: recv got EOF from 127.0.0.1:38048 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 19:32:46.067796 23771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.057483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.1.39.62:42219)
0926 19:32:46.057700 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.057707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.064947 (+ 7240us) server_negotiation.cc:407] Connection header received
0926 19:32:46.065588 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.065595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.065687 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.065797 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.067484 (+ 1687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34032: BlockingRecv error: recv got EOF from 127.0.0.1:34032 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 19:32:46.069568 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.051188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.1.39.60:34005)
0926 19:32:46.051386 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.051393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.064368 (+ 12975us) server_negotiation.cc:407] Connection header received
0926 19:32:46.066046 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.066052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.066145 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.066224 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.069220 (+ 2996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60746: BlockingRecv error: recv got EOF from 127.0.0.1:60746 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:32:46.135016 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.113579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.1.39.61:44881)
0926 19:32:46.113784 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.113791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.126100 (+ 12309us) server_negotiation.cc:407] Connection header received
0926 19:32:46.126713 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.126721 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.126989 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.127137 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.127931 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.129228 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.131571 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.132358 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.132487 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.133376 (+ 889us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.134124 (+ 748us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.134248 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.134528 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.134637 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 19:32:46.183883 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.169061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.1.39.60:34005)
0926 19:32:46.169269 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.169276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.181976 (+ 12700us) server_negotiation.cc:407] Connection header received
0926 19:32:46.182238 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.182245 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.182379 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.182501 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.183615 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60748: BlockingRecv error: recv got EOF from 127.0.0.1:60748 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 19:32:46.238127 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.216012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38060 (local address 127.1.39.61:44881)
0926 19:32:46.216279 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.216288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.231113 (+ 14825us) server_negotiation.cc:407] Connection header received
0926 19:32:46.231352 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.231358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.231455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.231549 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.232223 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.233477 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.235526 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.236217 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.236319 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.236942 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.237120 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.237189 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.237727 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.237818 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 19:32:46.257654 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.255110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.1.39.60:34005)
0926 19:32:46.255291 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.255298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.256036 (+ 738us) server_negotiation.cc:407] Connection header received
0926 19:32:46.256251 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.256258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.256368 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.256492 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.257416 (+ 924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60760: BlockingRecv error: recv got EOF from 127.0.0.1:60760 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:32:46.271879 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.264494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.1.39.61:44881)
0926 19:32:46.264672 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.264678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.265202 (+ 524us) server_negotiation.cc:407] Connection header received
0926 19:32:46.265424 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.265432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.265551 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.265660 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.266140 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.267336 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.269394 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.270238 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.270443 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.270932 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.271100 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.271205 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.271380 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.271477 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 19:32:46.285815 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.283655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.1.39.60:34005)
0926 19:32:46.283838 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.283844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.284338 (+ 494us) server_negotiation.cc:407] Connection header received
0926 19:32:46.284531 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.284538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.284646 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.284768 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.285576 (+ 808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60770: BlockingRecv error: recv got EOF from 127.0.0.1:60770 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:32:46.302270 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.295110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.1.39.61:44881)
0926 19:32:46.295340 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.295346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.295820 (+ 474us) server_negotiation.cc:407] Connection header received
0926 19:32:46.296025 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.296032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.296122 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.296228 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.296745 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.297972 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.299992 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.300807 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.300947 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.301445 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.301595 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.301690 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.301841 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.301931 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260926 19:32:46.311424 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.309204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.1.39.60:34005)
0926 19:32:46.309549 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.309558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.309951 (+ 393us) server_negotiation.cc:407] Connection header received
0926 19:32:46.310162 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.310167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.310250 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.310337 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.311171 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60780: BlockingRecv error: recv got EOF from 127.0.0.1:60780 (error 108)
Metrics: {"server-negotiator.queue_time_us":172}
I20260926 19:32:46.328328 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.322538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.1.39.61:44881)
0926 19:32:46.322750 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.322756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.323025 (+ 269us) server_negotiation.cc:407] Connection header received
0926 19:32:46.323189 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.323192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.323254 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.323329 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.323731 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.324470 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.326422 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.326809 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.326903 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.327478 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.327606 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.327691 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.327845 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.327912 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260926 19:32:46.336670 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.334438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.1.39.60:34005)
0926 19:32:46.334692 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.334698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.335441 (+ 743us) server_negotiation.cc:407] Connection header received
0926 19:32:46.335564 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.335570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.335669 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.335771 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.336449 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60792: BlockingRecv error: recv got EOF from 127.0.0.1:60792 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 19:32:46.361884 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.355972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.1.39.61:44881)
0926 19:32:46.356183 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.356191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.356576 (+ 385us) server_negotiation.cc:407] Connection header received
0926 19:32:46.356740 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.356746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.356842 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.356957 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.357281 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.358564 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.359900 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.360590 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.360735 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.361087 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.361211 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.361329 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.361497 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.361585 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 19:32:46.374073 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.372189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60808 (local address 127.1.39.60:34005)
0926 19:32:46.372409 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.372418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.372737 (+ 319us) server_negotiation.cc:407] Connection header received
0926 19:32:46.372867 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.372873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.372975 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.373087 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.373836 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60808: BlockingRecv error: recv got EOF from 127.0.0.1:60808 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 19:32:46.416612 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.410312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.1.39.61:44881)
0926 19:32:46.410595 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.410609 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.411046 (+ 437us) server_negotiation.cc:407] Connection header received
0926 19:32:46.411282 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.411288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.411398 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.411507 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.412016 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.412774 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.414759 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.415218 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.415309 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.415814 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.415988 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.416091 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.416245 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.416321 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260926 19:32:46.424631 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.422758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.1.39.60:34005)
0926 19:32:46.422968 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.422974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.423279 (+ 305us) server_negotiation.cc:407] Connection header received
0926 19:32:46.423479 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.423483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.423552 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.423632 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.424417 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60822: BlockingRecv error: recv got EOF from 127.0.0.1:60822 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 19:32:46.500603 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.492720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.1.39.61:44881)
0926 19:32:46.493018 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.493027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.493976 (+ 949us) server_negotiation.cc:407] Connection header received
0926 19:32:46.494342 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.494349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.494471 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.494560 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.495260 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.496572 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.498604 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.499290 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.499417 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.499870 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.500000 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.500085 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.500238 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.500318 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 19:32:46.511318 23768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:46.507722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60830 (local address 127.1.39.60:34005)
0926 19:32:46.507963 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.507973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.509881 (+ 1908us) server_negotiation.cc:407] Connection header received
0926 19:32:46.510081 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.510088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.510177 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.510292 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:46.511108 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60830: BlockingRecv error: recv got EOF from 127.0.0.1:60830 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 19:32:46.615513 23785 catalog_manager.cc:1259] Loaded cluster ID: 17ca343dc66f43728ceb7d38654c1c0c
I20260926 19:32:46.615619 23785 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: loading cluster ID for follower catalog manager: success
I20260926 19:32:46.618106 23785 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: acquiring CA information for follower catalog manager: success
I20260926 19:32:46.619835 23785 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02c1b9f194b74c7dbd0f51f3e93691d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:32:46.648495 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.643407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.1.39.61:44881)
0926 19:32:46.643655 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.643661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.643911 (+ 250us) server_negotiation.cc:407] Connection header received
0926 19:32:46.644144 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.644149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.644218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.644317 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.644727 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.645520 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.646846 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.647274 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.647407 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.647808 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.647909 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.648004 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.648151 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.648231 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 19:32:46.659399 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.654092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.1.39.60:34005)
0926 19:32:46.654334 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.654340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.654579 (+ 239us) server_negotiation.cc:407] Connection header received
0926 19:32:46.654751 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.654755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.654905 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.655000 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.655367 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.656083 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.657345 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.657778 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.657878 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.658309 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.658842 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.658949 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.659041 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.659104 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 19:32:46.678200 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.665361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.1.39.61:44881)
0926 19:32:46.665841 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.665847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.671754 (+ 5907us) server_negotiation.cc:407] Connection header received
0926 19:32:46.671956 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.671965 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.672094 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.672203 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.673142 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.674296 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.676406 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.676823 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.676892 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.677465 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.677583 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.677677 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.677821 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.677899 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260926 19:32:46.680274 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.665240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60838 (local address 127.1.39.60:34005)
0926 19:32:46.665432 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.665438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.673836 (+ 8398us) server_negotiation.cc:407] Connection header received
0926 19:32:46.673988 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.673994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.674073 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.674161 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.675045 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.676177 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.678264 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.678885 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.679035 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.679478 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.679590 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.679650 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.679860 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.679927 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:32:46.680500 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:46.665543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.1.39.62:42219)
0926 19:32:46.673435 (+ 7892us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:46.673443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:46.673465 (+ 22us) server_negotiation.cc:407] Connection header received
0926 19:32:46.673590 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:46.673595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:46.673847 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:46.674020 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:46.675085 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.675818 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.677969 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:46.678591 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:46.678730 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:46.679414 (+ 684us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:46.679884 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:46.679985 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:46.680126 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 19:32:46.680190 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7730,"thread_start_us":7632,"threads_started":1}
I20260926 19:32:46.685225 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23550
I20260926 19:32:46.686312 23612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:46.797354 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23550
I20260926 19:32:46.819439 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23624
I20260926 19:32:46.820556 23686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:46.927665 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23624
I20260926 19:32:46.948341 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23698
I20260926 19:32:46.949427 23760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:47.056404 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23698
2026-09-26T19:32:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4426 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 19:32:47 dist-test-slave-9wb0 krb5kdc[23949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 19:32:47 dist-test-slave-9wb0 krb5kdc[23949](info): set up 2 sockets
Sep 26 19:32:47 dist-test-slave-9wb0 krb5kdc[23949](info): commencing operation
krb5kdc: starting...
W20260926 19:32:48.177935 1180 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/test-user.keytab.
Sep 26 19:32:48 dist-test-slave-9wb0 krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T19:32:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T19:32:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.39.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
Entry for principal kudu/127.1.39.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab.
I20260926 19:32:48.467047 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38071
--webserver_interface=127.1.39.62
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 19:32:48.796685 23962 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:48.797021 23962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:48.797138 23962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:48.797266 23962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:48.804461 23962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:48.804550 23962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:48.804646 23962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:48.804733 23962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:48.804840 23962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:48.814590 23962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.62.keytab
--principal=kudu/127.1.39.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.39.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:48.815846 23962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:48.817123 23962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:48.828488 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:48.828583 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:48.828672 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:48.829656 23962 server_base.cc:1034] running on GCE node
I20260926 19:32:48.830389 23962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:48.831777 23962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:48.832971 23962 hybrid_clock.cc:648] HybridClock initialized: now 1790451168832911 us; error 39 us; skew 500 ppm
I20260926 19:32:48.833359 23962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:48 dist-test-slave-9wb0 krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:48.843503 23962 init.cc:377] Logged in from keytab as kudu/127.1.39.62@KRBTEST.COM (short username kudu)
I20260926 19:32:48.845070 23962 webserver.cc:460] Webserver started at http://127.1.39.62:36567/ using document root <none> and password file <none>
I20260926 19:32:48.845808 23962 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:48.845898 23962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:48.846236 23962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:48.849193 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data/instance:
uuid: "835c75d829c447dbba8f329cee4798e3"
format_stamp: "Formatted at 2026-09-26 19:32:48 on dist-test-slave-9wb0"
server_key: "59cbcd55c71bcc9b8995c4bea12ac8bc"
server_key_iv: "1b10ba27f589c29a7621e717170aa80d"
server_key_version: "encryptionkey@0"
I20260926 19:32:48.850009 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance:
uuid: "835c75d829c447dbba8f329cee4798e3"
format_stamp: "Formatted at 2026-09-26 19:32:48 on dist-test-slave-9wb0"
server_key: "59cbcd55c71bcc9b8995c4bea12ac8bc"
server_key_iv: "1b10ba27f589c29a7621e717170aa80d"
server_key_version: "encryptionkey@0"
I20260926 19:32:48.855170 23962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:32:48.858557 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:48.859910 23962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260926 19:32:48.860093 23962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal
uuid: "835c75d829c447dbba8f329cee4798e3"
format_stamp: "Formatted at 2026-09-26 19:32:48 on dist-test-slave-9wb0"
server_key: "59cbcd55c71bcc9b8995c4bea12ac8bc"
server_key_iv: "1b10ba27f589c29a7621e717170aa80d"
server_key_version: "encryptionkey@0"
I20260926 19:32:48.860304 23962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:48.882166 23962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:48.886130 23962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:48.886411 23962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:48.905261 23962 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.62:38071
I20260926 19:32:48.905300 24028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.62:38071 every 8 connection(s)
I20260926 19:32:48.906611 23962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/data/info.pb
I20260926 19:32:48.910493 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:48.915439 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 23962
I20260926 19:32:48.915668 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-0/wal/instance
I20260926 19:32:48.916265 1180 external_mini_cluster.cc:1428] Setting key 73e1e77fed31e6b1a3bfee948b00e296
I20260926 19:32:48.919011 24029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:48 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:48.944916 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:48.923186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38071 (local address 127.0.0.1:38384)
0926 19:32:48.924071 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:48.924130 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:48.926191 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:48.926984 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:48.926990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:48.927708 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:48.928035 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:48.928051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:48.929649 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:48.929657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:48.931581 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:48.931599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:48.932990 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:48.932996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:48.933190 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:48.933955 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:48.933979 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:48.936288 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:48.940769 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:48.940780 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:48.940789 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:48.941223 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:48.941723 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:48.941726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:48.941729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:48.941920 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:48.942912 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:48.942918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:48.943110 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 19:32:48.943422 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 19:32:48.943827 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":41,"thread_start_us":422,"threads_started":1}
I20260926 19:32:48.944967 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:48.923195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.1.39.62:38071)
0926 19:32:48.924880 (+ 1685us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:48.924892 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:48.924945 (+ 53us) server_negotiation.cc:407] Connection header received
0926 19:32:48.926539 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:48.926544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:48.926773 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:48.926967 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:48.928236 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:48.929497 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:48.931751 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:48.932735 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:48.932925 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:48.936530 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:48.936543 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:48.936560 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:48.936610 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:48.940628 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:48.941382 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:48.941394 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:48.941400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:48.941604 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:48.942052 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:48.942060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:48.942064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:48.942764 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:48.942968 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:48.943629 (+ 661us) server_negotiation.cc:299] Negotiation successful
0926 19:32:48.944004 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":1466,"thread_start_us":1267,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.61@KRBTEST.COM; defaulting to no policy
I20260926 19:32:48.948164 24029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:48.949496 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:48.948964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58768)
0926 19:32:48.949139 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:48.949273 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 19:32:48.949765 23981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.61@KRBTEST.COM" created.
I20260926 19:32:48.951352 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:48.951022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58784)
0926 19:32:48.951169 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:48.951245 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 19:32:48.951527 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:40483: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
Entry for principal kudu/127.1.39.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab.
I20260926 19:32:48.984772 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:40483
--webserver_interface=127.1.39.61
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:32:49.000888 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } attempt: 1
I20260926 19:32:49.002276 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.001692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58792)
0926 19:32:49.001947 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.002093 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 19:32:49.003734 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.003118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58800)
0926 19:32:49.003354 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.003490 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 19:32:49.003949 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:40483: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
I20260926 19:32:49.072229 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } attempt: 2
I20260926 19:32:49.073554 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.072926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58804)
0926 19:32:49.073210 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.073377 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 19:32:49.074754 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.074234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58814)
0926 19:32:49.074426 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.074544 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 19:32:49.074958 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:40483: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
I20260926 19:32:49.166224 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } attempt: 3
I20260926 19:32:49.167506 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.166975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58822)
0926 19:32:49.167205 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.167331 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 19:32:49.168825 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.168330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58824)
0926 19:32:49.168521 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.168629 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:32:49.169030 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:40483: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
I20260926 19:32:49.312300 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } attempt: 4
I20260926 19:32:49.313552 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.312988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58838)
0926 19:32:49.313326 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.313422 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 19:32:49.314594 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.314279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58846)
0926 19:32:49.314408 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.314484 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 19:32:49.314859 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.61:40483: Network error: Client connection negotiation failed: client connection to 127.1.39.61:40483: connect: Connection refused (error 111)
W20260926 19:32:49.335922 24036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:49.336273 24036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:49.336387 24036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:49.336517 24036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 19:32:49.343690 24036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:49.343777 24036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:49.343876 24036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:49.343966 24036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:49.344082 24036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:49.354079 24036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.61.keytab
--principal=kudu/127.1.39.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.61:40483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.39.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:49.355365 24036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:49.356660 24036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:49.368283 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:49.368362 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:49.368330 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:49.369328 24036 server_base.cc:1034] running on GCE node
I20260926 19:32:49.370154 24036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:49.371636 24036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:49.372881 24036 hybrid_clock.cc:648] HybridClock initialized: now 1790451169372803 us; error 53 us; skew 500 ppm
I20260926 19:32:49.373207 24036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:49 dist-test-slave-9wb0 krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:49.376291 24036 init.cc:377] Logged in from keytab as kudu/127.1.39.61@KRBTEST.COM (short username kudu)
I20260926 19:32:49.377779 24036 webserver.cc:460] Webserver started at http://127.1.39.61:36477/ using document root <none> and password file <none>
I20260926 19:32:49.378460 24036 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:49.378595 24036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:49.378908 24036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:49.381881 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data/instance:
uuid: "beca67a90f7b4c5b98c6ecd61e439e79"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "0513153ee7cd3c7e1c45c3fb30f8ad59"
server_key_iv: "c0aff7a5092730cfd0574348830809d4"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.382663 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal/instance:
uuid: "beca67a90f7b4c5b98c6ecd61e439e79"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "0513153ee7cd3c7e1c45c3fb30f8ad59"
server_key_iv: "c0aff7a5092730cfd0574348830809d4"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.387797 24036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 19:32:49.391330 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:49.392678 24036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 19:32:49.392859 24036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal
uuid: "beca67a90f7b4c5b98c6ecd61e439e79"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "0513153ee7cd3c7e1c45c3fb30f8ad59"
server_key_iv: "c0aff7a5092730cfd0574348830809d4"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.393076 24036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:49.431572 24036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:49.435928 24036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:49.436234 24036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:49.455173 24036 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.61:40483
I20260926 19:32:49.455193 24102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.61:40483 every 8 connection(s)
I20260926 19:32:49.456547 24036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/data/info.pb
I20260926 19:32:49.460458 24103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:49.464089 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 24036
I20260926 19:32:49.464295 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-1/wal/instance
I20260926 19:32:49.464905 1180 external_mini_cluster.cc:1428] Setting key 2f393f14cde71654366fe9d11ad28773
I20260926 19:32:49.469795 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:49 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:49.494268 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.473584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.1.39.62:38071)
0926 19:32:49.474245 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:49.474253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:49.475447 (+ 1194us) server_negotiation.cc:407] Connection header received
0926 19:32:49.478468 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:49.478473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:49.478578 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:49.478695 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:49.480265 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.481065 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.482660 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.483071 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.483207 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.487199 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:49.487204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:49.487208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:49.487245 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:49.490572 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.491485 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.491491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.491496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.491694 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.492125 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.492131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.492134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.492485 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:49.492645 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:49.493651 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0926 19:32:49.493881 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":370,"threads_started":1}
I20260926 19:32:49.495263 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.474340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38071 (local address 127.0.0.1:38392)
0926 19:32:49.475250 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.475305 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:49.478126 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:49.478950 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:49.478960 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:49.479489 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:49.480031 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.480048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.481228 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.481233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.482504 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.482516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.483294 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.483301 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.483538 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.484363 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:49.484390 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:49.486949 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:49.490731 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.490769 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.490789 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.491339 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.491807 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.491812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.491815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.492004 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.492655 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:49.492666 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:49.493204 (+ 538us) client_negotiation.cc:769] Sending connection context
0926 19:32:49.493556 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 19:32:49.494098 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":358,"threads_started":1}
WARNING: no policy specified for kudu/127.1.39.60@KRBTEST.COM; defaulting to no policy
I20260926 19:32:49.497548 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.39.60@KRBTEST.COM" created.
Sep 26 19:32:49 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:49.512542 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.498451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58860)
0926 19:32:49.498705 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.498726 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:49.498843 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:49.499837 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:49.499841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:49.500275 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:49.500396 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.500404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.501859 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.501866 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.502709 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.502719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.503518 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.503524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.503751 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.504335 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:49.504353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:49.506307 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:49.510061 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.510068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.510073 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.510434 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.510815 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.510820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.510823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.510955 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.511534 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:49.511543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:49.511689 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 19:32:49.511832 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 19:32:49.512110 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":23}
I20260926 19:32:49.512578 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.498718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58860 (local address 127.1.39.61:40483)
0926 19:32:49.499427 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:49.499438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:49.499463 (+ 25us) server_negotiation.cc:407] Connection header received
0926 19:32:49.499507 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:49.499514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:49.499688 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:49.499837 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:49.500558 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.501721 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.502838 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.503373 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.503503 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.506512 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:49.506523 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:49.506536 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:49.506567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:49.509934 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.510573 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.510584 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.510588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.510706 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.511064 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.511068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.511072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.511406 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:49.511619 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:49.511970 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 19:32:49.512210 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":556,"thread_start_us":403,"threads_started":1}
I20260926 19:32:49.515352 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:49.516680 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.516090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54888)
0926 19:32:49.516295 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.516463 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 19:32:49.516949 24055 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.518667 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.518205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54902)
0926 19:32:49.518400 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.518511 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 19:32:49.518898 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
Entry for principal kudu/127.1.39.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab.
I20260926 19:32:49.534889 1180 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksjShy7/build/asan/bin/kudu
/tmp/dist-test-tasksjShy7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:43629
--webserver_interface=127.1.39.60
--webserver_port=0
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.39.20:33209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 19:32:49.568248 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 1
I20260926 19:32:49.569460 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.568945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54904)
0926 19:32:49.569151 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.569260 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 19:32:49.570452 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.570059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54906)
0926 19:32:49.570222 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.570319 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 19:32:49.570727 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.614135 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } attempt: 5
Sep 26 19:32:49 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:49.629699 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.615184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.1.39.61:40483)
0926 19:32:49.615381 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:49.615386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:49.615406 (+ 20us) server_negotiation.cc:407] Connection header received
0926 19:32:49.615544 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:49.615548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:49.615628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:49.615711 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:49.616843 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.617753 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.619646 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.620121 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.620235 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.624218 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:49.624224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:49.624228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:49.624272 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:49.626981 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.627799 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.627804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.627806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.627925 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:49.628320 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:49.628323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:49.628325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:49.628605 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:49.628719 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:49.629215 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 19:32:49.629476 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 19:32:49.629977 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.615019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58868)
0926 19:32:49.615249 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.615272 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:49.615409 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:49.615778 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:49.615784 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:49.616348 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:49.616629 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.616647 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.617932 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.617940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.619461 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:49.619478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:49.620281 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:49.620287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:49.620514 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:49.621357 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:49.621381 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:49.623948 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:49.627132 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.627138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.627141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.627683 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.628043 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:49.628048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:49.628051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:49.628219 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:49.628721 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:49.628725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:49.628893 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 19:32:49.629092 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 19:32:49.629508 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 19:32:49.631198 24029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:49.632272 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.631832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54908)
0926 19:32:49.632016 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.632108 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 19:32:49.633375 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.632956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54918)
0926 19:32:49.633124 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.633231 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 19:32:49.633596 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.638938 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 2
I20260926 19:32:49.639919 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.639455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54930)
0926 19:32:49.639642 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.639750 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 19:32:49.640900 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.640539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54934)
0926 19:32:49.640681 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.640763 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:32:49.641160 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.684867 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 1
I20260926 19:32:49.686249 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.685753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54942)
0926 19:32:49.685973 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.686071 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 19:32:49.687398 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.687063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54950)
0926 19:32:49.687221 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.687295 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 19:32:49.687755 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.732398 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 3
I20260926 19:32:49.733911 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.733122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54960)
0926 19:32:49.733540 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.733675 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 19:32:49.735309 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.734727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54970)
0926 19:32:49.735010 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.735127 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260926 19:32:49.735597 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.755986 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 2
I20260926 19:32:49.757179 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.756705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54980)
0926 19:32:49.756928 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.757021 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 19:32:49.758169 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.757847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54986)
0926 19:32:49.757994 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.758071 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 19:32:49.758347 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.864593 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 3
I20260926 19:32:49.865782 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.865200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:54998)
0926 19:32:49.865452 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.865581 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 19:32:49.866875 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.866473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55000)
0926 19:32:49.866654 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.866742 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 19:32:49.867094 24029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
I20260926 19:32:49.878830 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 4
I20260926 19:32:49.879966 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.879435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55006)
0926 19:32:49.879674 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.879773 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 19:32:49.880005 24111 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 19:32:49.880414 24111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 19:32:49.880535 24111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 19:32:49.880663 24111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 19:32:49.881019 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:49.880692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55016)
0926 19:32:49.880850 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:49.880915 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 19:32:49.881237 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.39.60:43629: Network error: Client connection negotiation failed: client connection to 127.1.39.60:43629: connect: Connection refused (error 111)
W20260926 19:32:49.888032 24111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 19:32:49.888118 24111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 19:32:49.888218 24111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 19:32:49.888291 24111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 19:32:49.888391 24111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 19:32:49.898591 24111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.39.20:33209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.39.62:38071,127.1.39.61:40483,127.1.39.60:43629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/krb5kdc/kudu_127.1.39.60.keytab
--principal=kudu/127.1.39.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.39.60:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.39.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 18:32:33 UTC on ccycloud.quasar-tsfqhi.root.comops.site
build id 4675021
ASAN enabled
I20260926 19:32:49.899863 24111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 19:32:49.901167 24111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 19:32:49.914059 24117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 19:32:49.914675 24116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:49.914927 24111 server_base.cc:1034] running on GCE node
W20260926 19:32:49.915467 24119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 19:32:49.916429 24111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 19:32:49.917886 24111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 19:32:49.919117 24111 hybrid_clock.cc:648] HybridClock initialized: now 1790451169919059 us; error 41 us; skew 500 ppm
I20260926 19:32:49.919430 24111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 19:32:49 dist-test-slave-9wb0 krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 19:32:49.922617 24111 init.cc:377] Logged in from keytab as kudu/127.1.39.60@KRBTEST.COM (short username kudu)
I20260926 19:32:49.924057 24111 webserver.cc:460] Webserver started at http://127.1.39.60:35397/ using document root <none> and password file <none>
I20260926 19:32:49.924733 24111 fs_manager.cc:359] Metadata directory not provided
I20260926 19:32:49.924844 24111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 19:32:49.925161 24111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 19:32:49.928092 24111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data/instance:
uuid: "7a40c8c0415242099eb7b3ab64c3721b"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "e3761ae59888213270c151a5b908c0df"
server_key_iv: "814eefa64fd6c04a11c512f1e3846622"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.928860 24111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal/instance:
uuid: "7a40c8c0415242099eb7b3ab64c3721b"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "e3761ae59888213270c151a5b908c0df"
server_key_iv: "814eefa64fd6c04a11c512f1e3846622"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.934049 24111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260926 19:32:49.937497 24126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 19:32:49.938902 24111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 19:32:49.939100 24111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data,/tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal
uuid: "7a40c8c0415242099eb7b3ab64c3721b"
format_stamp: "Formatted at 2026-09-26 19:32:49 on dist-test-slave-9wb0"
server_key: "e3761ae59888213270c151a5b908c0df"
server_key_iv: "814eefa64fd6c04a11c512f1e3846622"
server_key_version: "encryptionkey@0"
I20260926 19:32:49.939314 24111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 19:32:49.985734 24111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 19:32:49.990391 24111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 19:32:49.990684 24111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 19:32:50.009888 24111 rpc_server.cc:307] RPC server started. Bound to: 127.1.39.60:43629
I20260926 19:32:50.009930 24177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.39.60:43629 every 8 connection(s)
I20260926 19:32:50.011288 24111 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/data/info.pb
I20260926 19:32:50.014129 1180 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksjShy7/build/asan/bin/kudu as pid 24111
I20260926 19:32:50.014312 1180 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksjShy7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790450801920901-1180-0/minicluster-data/master-2/wal/instance
I20260926 19:32:50.014832 1180 external_mini_cluster.cc:1428] Setting key c95c30cfb2a20b185aeb7b8f9322eaf5
I20260926 19:32:50.015480 24178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 19:32:50.025611 24178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
I20260926 19:32:50.039079 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.016652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.1.39.60:43629)
0926 19:32:50.017490 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.017502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.017544 (+ 42us) server_negotiation.cc:407] Connection header received
0926 19:32:50.019078 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.019100 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.019595 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.019961 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.021784 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.023119 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.024969 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.025888 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.026032 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.031109 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.031117 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.031127 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.031166 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.034577 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.035360 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.035367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.035373 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.035539 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.035987 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.035991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.035993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.036526 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.036775 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.037361 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.037840 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":366,"threads_started":1}
I20260926 19:32:50.044305 24029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 4
I20260926 19:32:50.045122 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.028751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.62:38071 (local address 127.0.0.1:38408)
0926 19:32:50.029615 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:50.029661 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:50.029833 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:50.030769 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:50.030775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:50.031447 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:50.031780 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.031794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.032857 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.032865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.034221 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.034236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.035154 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.035162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.035375 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.035954 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:50.035981 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:50.038437 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:50.042183 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.042197 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.042207 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.042855 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.043266 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.043271 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.043275 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.043439 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.043987 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:50.043995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:50.044167 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 19:32:50.044355 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 19:32:50.044677 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":368,"threads_started":1}
I20260926 19:32:50.045255 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.028853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.1.39.62:38071)
0926 19:32:50.030367 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.030375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.030396 (+ 21us) server_negotiation.cc:407] Connection header received
0926 19:32:50.030459 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.030464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.030564 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.030690 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.031938 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.032712 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.034375 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.034940 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.035035 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.038639 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.038646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.038650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.038694 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.042048 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.042996 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.043006 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.043008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.043149 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.043557 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.043562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.043565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.043833 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.043987 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.044513 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.044748 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1368,"thread_start_us":1200,"threads_started":1}
I20260926 19:32:50.046775 24178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.62@KRBTEST.COM
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.62@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:50.060767 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.045655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55024 (local address 127.1.39.60:43629)
0926 19:32:50.045806 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.045810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.045823 (+ 13us) server_negotiation.cc:407] Connection header received
0926 19:32:50.045858 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.045861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.045936 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.046033 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.046886 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.047908 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.049644 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.050252 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.050374 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.054713 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.054720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.054727 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.054769 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.057753 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.058452 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.058457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.058462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.058652 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.059112 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.059120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.059125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.059492 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.059636 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.060165 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.060393 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 19:32:50.060786 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.044870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55024)
0926 19:32:50.045224 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:50.045248 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:50.045367 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:50.046077 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:50.046082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:50.046568 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:50.046759 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.046769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.048028 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.048033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.049487 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.049501 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.050419 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.050424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.050633 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.051228 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:50.051256 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:50.054500 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:50.057888 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.057894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.057898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.058327 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.058779 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.058784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.058788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.058970 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.059702 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:50.059708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:50.059862 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 19:32:50.060072 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 19:32:50.060412 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"spinlock_wait_cycles":21248}
I20260926 19:32:50.061647 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.047627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.61:40483 (local address 127.0.0.1:58878)
0926 19:32:50.048072 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:50.048085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:50.048186 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:50.048486 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:50.048490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:50.048867 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:50.048998 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.049005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.050311 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.050319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.051429 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.051440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.052134 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.052140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.052315 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.052832 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:50.052848 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:50.054904 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:50.058858 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.058866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.058872 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.059403 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.059808 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.059813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.059818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.059974 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.060678 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:50.060682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:50.060848 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 19:32:50.060980 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 19:32:50.061246 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368}
I20260926 19:32:50.062202 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.042393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.1.39.62:38071)
0926 19:32:50.043042 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.043047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.043070 (+ 23us) server_negotiation.cc:407] Connection header received
0926 19:32:50.043110 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.043114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.043172 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.043281 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.044960 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.046741 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.048491 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.049202 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.049361 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.054862 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.054867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.054870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.054899 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.058412 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.059148 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.059155 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.059161 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.059309 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.059736 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.059742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.059745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.060138 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.060401 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.061577 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.061804 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":266,"threads_started":1}
I20260926 19:32:50.062578 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.047613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.1.39.61:40483)
0926 19:32:50.047781 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.047787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.048155 (+ 368us) server_negotiation.cc:407] Connection header received
0926 19:32:50.048285 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.048290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.048362 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.048464 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.049144 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.050105 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.051584 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.051985 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.052119 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.055109 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.055115 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.055120 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.055159 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.058685 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.059498 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.059504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.059508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.059651 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.060109 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.060114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.060117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.060468 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.060615 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.061994 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.062218 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:32:50.063500 24178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260926 19:32:50.068090 24029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: Bootstrap starting.
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.60@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:32:50.072348 24029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: Neither blocks nor log segments found. Creating new log.
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.39.61@KRBTEST.COM
I20260926 19:32:50.073789 24029 log.cc:824] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:50.077171 24029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: No bootstrap required, opened a new log
I20260926 19:32:50.080880 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.064499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55026)
0926 19:32:50.064986 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:50.065003 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:50.065115 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:50.065479 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:50.065484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:50.065938 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:50.066127 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.066138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.067395 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.067401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.068612 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.068624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.069498 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.069504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.069814 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.070392 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:50.070416 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:50.073669 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:50.078105 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.078111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.078114 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.078642 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.079039 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.079044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.079048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.079177 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.079846 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:50.079850 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:50.079991 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 19:32:50.080151 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 19:32:50.080449 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":22}
I20260926 19:32:50.081090 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.064179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.1.39.61:40483)
0926 19:32:50.064335 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.064356 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.064373 (+ 17us) server_negotiation.cc:407] Connection header received
0926 19:32:50.064417 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.064421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.064485 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.064589 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.066003 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.066728 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.068236 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.068860 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.068985 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.073989 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.073997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.074002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.074040 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.077945 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.078674 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.078680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.078683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.078855 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.079280 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.079287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.079292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.079645 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.079917 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.080225 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.080442 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 19:32:50.081768 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.064801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.1.39.60:43629)
0926 19:32:50.064996 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.065004 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.065051 (+ 47us) server_negotiation.cc:407] Connection header received
0926 19:32:50.065227 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.065235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.065341 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.065451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.066270 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.067256 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.068737 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.069338 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.069478 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.073882 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.073890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.073895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.073933 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.077975 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.078757 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.078764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.078770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.078911 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.079311 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.079315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.079318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.079735 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.081124 (+ 1389us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.081279 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.081505 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":53}
I20260926 19:32:50.085888 24029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.086184 24029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835c75d829c447dbba8f329cee4798e3, State: Initialized, Role: FOLLOWER
I20260926 19:32:50.086565 24178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: Bootstrap starting.
I20260926 19:32:50.086648 24029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.088064 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.088613 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.088897 1180 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 19:32:50.089408 24029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:50.090334 24178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:50.091496 24178 log.cc:824] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:50.092279 24004 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "835c75d829c447dbba8f329cee4798e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38414
I20260926 19:32:50.092746 24004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 19:32:50.092996 24004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:50.094631 24178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: No bootstrap required, opened a new log
I20260926 19:32:50.097477 24004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
W20260926 19:32:50.103513 24198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:32:50.103730 24198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:32:50.104122 24004 leader_election.cc:288] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [CANDIDATE]: Term 1 election: Requested vote from peers beca67a90f7b4c5b98c6ecd61e439e79 (127.1.39.61:40483), 7a40c8c0415242099eb7b3ab64c3721b (127.1.39.60:43629)
I20260926 19:32:50.105839 24178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.106297 24178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a40c8c0415242099eb7b3ab64c3721b, State: Initialized, Role: FOLLOWER
I20260926 19:32:50.106877 24178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.103288 23962 master_runner.cc:186] Error getting master registration for 127.1.39.61:40483: OK, instance_id { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" instance_seqno: 1790451169449753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:32:50.114672 23962 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 19:32:50.116954 24178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:50.115382 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.117228 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.117020 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835c75d829c447dbba8f329cee4798e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a40c8c0415242099eb7b3ab64c3721b"
I20260926 19:32:50.117579 24153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:50.121265 24153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835c75d829c447dbba8f329cee4798e3 in term 1.
I20260926 19:32:50.122042 23981 leader_election.cc:302] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a40c8c0415242099eb7b3ab64c3721b, 835c75d829c447dbba8f329cee4798e3; no voters:
I20260926 19:32:50.122574 24186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 19:32:50.133783 24186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [term 1 LEADER]: Becoming Leader. State: Replica: 835c75d829c447dbba8f329cee4798e3, State: Running, Role: LEADER
I20260926 19:32:50.134696 24186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
W20260926 19:32:50.139333 23983 tablet.cc:2369] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 19:32:50.141713 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835c75d829c447dbba8f329cee4798e3. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.142158 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:50.158946 24208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 19:32:50.161428 24208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 19:32:50.180478 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } attempt: 5
I20260926 19:32:50.203776 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.165614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.1.39.62:38071)
0926 19:32:50.165947 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.165957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.170109 (+ 4152us) server_negotiation.cc:407] Connection header received
0926 19:32:50.171545 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.171551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.171640 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.171729 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.178049 (+ 6320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.179225 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.184898 (+ 5673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.185711 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.185840 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.192605 (+ 6765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.192614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.192620 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.192666 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.197473 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.198851 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.198858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.198863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.199023 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.201706 (+ 2683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.201713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.201718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.202151 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.202458 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.203080 (+ 622us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.203361 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 19:32:50.218089 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.170492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.1.39.61:40483)
0926 19:32:50.170723 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.170731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.181774 (+ 11043us) server_negotiation.cc:407] Connection header received
0926 19:32:50.182353 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.182359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.182438 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.182512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.186736 (+ 4224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.187851 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.193541 (+ 5690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.194238 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.194350 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.207274 (+ 12924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.207282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.207287 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.207332 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.211467 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.215544 (+ 4077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.215550 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.215553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.215714 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.216224 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.216230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.216233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.216605 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.216849 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.217472 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.217726 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 19:32:50.230803 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.181882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55044 (local address 127.1.39.60:43629)
0926 19:32:50.182072 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.182079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.205189 (+ 23110us) server_negotiation.cc:407] Connection header received
0926 19:32:50.205458 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.205464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.205543 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.205629 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.212399 (+ 6770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.213633 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.219161 (+ 5528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.219896 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.220006 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.223595 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.223602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.223607 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.223651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.227364 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.228317 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.228323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.228326 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.228479 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.228925 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.228930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.228934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.229362 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.229616 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.230178 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.230437 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 19:32:50.232558 24223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 19:32:50.232730 24223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 19:32:50.235323 24153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [term 1 FOLLOWER]: Refusing update from remote peer 835c75d829c447dbba8f329cee4798e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:50.236994 24204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:50.209957 24111 master_runner.cc:186] Error getting master registration for 127.1.39.61:40483: OK, instance_id { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" instance_seqno: 1790451169449753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 19:32:50.241350 24111 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 19:32:50.242856 24132 tablet.cc:2369] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 19:32:50 dist-test-slave-9wb0 krb5kdc[23949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790451169, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.39.61@KRBTEST.COM for kudu/127.1.39.60@KRBTEST.COM
I20260926 19:32:50.253867 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835c75d829c447dbba8f329cee4798e3. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.254169 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.256644 24213 mvcc.cc:205] Tried to move back new op lower bound from 7333687993237602304 to 7333687992878153728. Current Snapshot: MvccSnapshot[applied={T|T < 7333687993237602304}]
I20260926 19:32:50.266675 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.266916 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.268090 24186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.268399 24186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:50.270118 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.181262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.39.60:43629 (local address 127.0.0.1:55030)
0926 19:32:50.181677 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 19:32:50.181699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 19:32:50.181806 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 19:32:50.203649 (+ 21843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 19:32:50.203655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 19:32:50.204258 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0926 19:32:50.204552 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.204569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.234423 (+ 29854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.234434 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.236436 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 19:32:50.236455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.241647 (+ 5192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.241657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 19:32:50.241961 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.242930 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 19:32:50.242996 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0926 19:32:50.246582 (+ 3586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 19:32:50.265717 (+ 19135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.265724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.265729 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.266302 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.267593 (+ 1291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 19:32:50.267600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 19:32:50.267605 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 19:32:50.267806 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 19:32:50.269018 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 19:32:50.269024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 19:32:50.269198 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 19:32:50.269377 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 19:32:50.269715 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260926 19:32:50.271090 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.181980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.1.39.60:43629)
0926 19:32:50.201347 (+ 19367us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.201357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.201389 (+ 32us) server_negotiation.cc:407] Connection header received
0926 19:32:50.201463 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.201471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.201572 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.221404 (+ 19832us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.221731 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.223291 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.236687 (+ 13396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.237604 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.245112 (+ 7508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.247753 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 19:32:50.247761 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 19:32:50.247766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 19:32:50.247876 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0926 19:32:50.264661 (+ 16785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.266711 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.266715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.266718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.267051 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 19:32:50.268011 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 19:32:50.268018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 19:32:50.268021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 19:32:50.268891 (+ 870us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 19:32:50.269029 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.269638 (+ 609us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.270125 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19177,"thread_start_us":19707,"threads_started":1}
I20260926 19:32:50.273172 24208 catalog_manager.cc:1347] Generated new cluster ID: b0ea64cad21a47a1bf5d9c2a368f747e
I20260926 19:32:50.273275 24208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 19:32:50.279979 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: Bootstrap starting.
I20260926 19:32:50.287484 24103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: Neither blocks nor log segments found. Creating new log.
I20260926 19:32:50.289947 24103 log.cc:824] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: Log is configured to *not* fsync() on all Append() calls
I20260926 19:32:50.305481 24208 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 19:32:50.306422 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: No bootstrap required, opened a new log
I20260926 19:32:50.311056 24208 catalog_manager.cc:1504] Loading token signing keys...
I20260926 19:32:50.328539 24103 raft_consensus.cc:348] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.329334 24103 raft_consensus.cc:729] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beca67a90f7b4c5b98c6ecd61e439e79, State: Initialized, Role: FOLLOWER
I20260926 19:32:50.330485 24208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3: Generated new TSK 0
I20260926 19:32:50.331967 24208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 19:32:50.331621 24103 consensus_queue.cc:260] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } }
I20260926 19:32:50.341778 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.342355 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.344177 24078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "835c75d829c447dbba8f329cee4798e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "beca67a90f7b4c5b98c6ecd61e439e79"
I20260926 19:32:50.344862 24103 sys_catalog.cc:564] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 19:32:50.345041 24077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 19:32:50.348980 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [term 1 FOLLOWER]: Refusing update from remote peer 835c75d829c447dbba8f329cee4798e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 19:32:50.352363 24078 raft_consensus.cc:2454] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835c75d829c447dbba8f329cee4798e3 in term 1.
I20260926 19:32:50.355772 24204 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 19:32:50.422068 24230 sys_catalog.cc:455] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835c75d829c447dbba8f329cee4798e3. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.422436 24230 sys_catalog.cc:458] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.422696 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.423095 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 19:32:50.496681 24251 catalog_manager.cc:1259] Loaded cluster ID: b0ea64cad21a47a1bf5d9c2a368f747e
I20260926 19:32:50.497370 24251 catalog_manager.cc:1552] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: loading cluster ID for follower catalog manager: success
W20260926 19:32:50.499838 24184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.481634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.1.39.62:38071)
0926 19:32:50.482004 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.482011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.496835 (+ 14824us) server_negotiation.cc:407] Connection header received
0926 19:32:50.497691 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.497697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.497788 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.497866 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.499550 (+ 1684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38430: BlockingRecv error: recv got EOF from 127.0.0.1:38430 (error 108)
Metrics: {"server-negotiator.queue_time_us":233}
W20260926 19:32:50.501266 24108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.490346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.1.39.61:40483)
0926 19:32:50.490556 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.490565 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.496931 (+ 6366us) server_negotiation.cc:407] Connection header received
0926 19:32:50.498167 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.498173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.498260 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.498333 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.501070 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58906: BlockingRecv error: recv got EOF from 127.0.0.1:58906 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 19:32:50.503808 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.495759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.1.39.60:43629)
0926 19:32:50.495931 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.495937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.502010 (+ 6073us) server_negotiation.cc:407] Connection header received
0926 19:32:50.502543 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.502550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.502652 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.502731 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.503581 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55048: BlockingRecv error: recv got EOF from 127.0.0.1:55048 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 19:32:50.512900 24251 catalog_manager.cc:1574] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: acquiring CA information for follower catalog manager: success
I20260926 19:32:50.516664 24251 catalog_manager.cc:1602] T 00000000000000000000000000000000 P beca67a90f7b4c5b98c6ecd61e439e79: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:32:50.555130 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.539871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58920 (local address 127.1.39.61:40483)
0926 19:32:50.540064 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.540071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.547047 (+ 6976us) server_negotiation.cc:407] Connection header received
0926 19:32:50.547612 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.547619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.547846 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.547984 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.548735 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.549932 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.552058 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.552737 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.552837 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.553635 (+ 798us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.554295 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.554411 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.554672 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.554770 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 19:32:50.595541 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.585646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.1.39.60:43629)
0926 19:32:50.585843 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.585850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.592873 (+ 7023us) server_negotiation.cc:407] Connection header received
0926 19:32:50.593722 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.593731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.593871 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.593962 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.595192 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55052: BlockingRecv error: recv got EOF from 127.0.0.1:55052 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:32:50.646026 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.631703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.1.39.61:40483)
0926 19:32:50.631894 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.631901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.638778 (+ 6877us) server_negotiation.cc:407] Connection header received
0926 19:32:50.639299 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.639306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.639398 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.639477 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.640186 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.641388 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.643645 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.644363 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.644456 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.645113 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.645276 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.645357 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.645600 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.645684 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260926 19:32:50.694607 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.676653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.1.39.60:43629)
0926 19:32:50.676859 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.676866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.689975 (+ 13109us) server_negotiation.cc:407] Connection header received
0926 19:32:50.690160 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.690166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.690280 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.690422 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.694339 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55058: BlockingRecv error: recv got EOF from 127.0.0.1:55058 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 19:32:50.749818 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.733265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.1.39.61:40483)
0926 19:32:50.733496 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.733503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.742077 (+ 8574us) server_negotiation.cc:407] Connection header received
0926 19:32:50.742718 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.742730 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.742856 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.742925 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.743967 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.745175 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.747372 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.748062 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.748166 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.748854 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.749024 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.749136 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.749326 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.749433 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 19:32:50.758953 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.756617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.1.39.60:43629)
0926 19:32:50.756799 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.756805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.757351 (+ 546us) server_negotiation.cc:407] Connection header received
0926 19:32:50.757573 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.757579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.757701 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.757825 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.758703 (+ 878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55062: BlockingRecv error: recv got EOF from 127.0.0.1:55062 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 19:32:50.772980 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.765930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.1.39.61:40483)
0926 19:32:50.766165 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.766171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.766689 (+ 518us) server_negotiation.cc:407] Connection header received
0926 19:32:50.766907 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.766913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.767012 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.767115 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.767750 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.768894 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.770906 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.771623 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.771748 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.772224 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.772364 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.772452 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.772616 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.772700 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260926 19:32:50.782538 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.780404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.1.39.60:43629)
0926 19:32:50.780686 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.780698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.781006 (+ 308us) server_negotiation.cc:407] Connection header received
0926 19:32:50.781211 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.781218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.781335 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.781444 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.782262 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55066: BlockingRecv error: recv got EOF from 127.0.0.1:55066 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 19:32:50.801766 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.794575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58942 (local address 127.1.39.61:40483)
0926 19:32:50.794823 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.794829 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.795337 (+ 508us) server_negotiation.cc:407] Connection header received
0926 19:32:50.795534 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.795540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.795631 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.795735 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.796307 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.797578 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.799520 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.800318 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.800447 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.800892 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.801056 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.801150 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.801331 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.801417 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260926 19:32:50.811388 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.808526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.1.39.60:43629)
0926 19:32:50.808718 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.808724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.809843 (+ 1119us) server_negotiation.cc:407] Connection header received
0926 19:32:50.810016 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.810023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.810124 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.810244 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.811145 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55070: BlockingRecv error: recv got EOF from 127.0.0.1:55070 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 19:32:50.836812 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.830727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.1.39.61:40483)
0926 19:32:50.830941 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.830950 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.831993 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 19:32:50.832209 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.832214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.832283 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.832379 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.832780 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.833664 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.835009 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.835529 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.835664 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.836079 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.836188 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.836285 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.836459 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.836553 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 19:32:50.846021 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.843873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.1.39.60:43629)
0926 19:32:50.844069 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.844076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.844558 (+ 482us) server_negotiation.cc:407] Connection header received
0926 19:32:50.844770 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.844775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.844847 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.844958 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.845799 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55076: BlockingRecv error: recv got EOF from 127.0.0.1:55076 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 19:32:50.874555 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "835c75d829c447dbba8f329cee4798e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835c75d829c447dbba8f329cee4798e3" member_type: VOTER last_known_addr { host: "127.1.39.62" port: 38071 } } peers { permanent_uuid: "beca67a90f7b4c5b98c6ecd61e439e79" member_type: VOTER last_known_addr { host: "127.1.39.61" port: 40483 } } peers { permanent_uuid: "7a40c8c0415242099eb7b3ab64c3721b" member_type: VOTER last_known_addr { host: "127.1.39.60" port: 43629 } } }
I20260926 19:32:50.874992 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835c75d829c447dbba8f329cee4798e3 [sys.catalog]: This master's current role is: LEADER
I20260926 19:32:50.887878 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.881523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58962 (local address 127.1.39.61:40483)
0926 19:32:50.881777 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.881784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.882142 (+ 358us) server_negotiation.cc:407] Connection header received
0926 19:32:50.882377 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.882381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.882464 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.882537 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.882996 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.883772 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.885802 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.886395 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.886536 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.887004 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.887120 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.887194 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.887401 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.887494 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 19:32:50.896275 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.894320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.1.39.60:43629)
0926 19:32:50.894501 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.894509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.894880 (+ 371us) server_negotiation.cc:407] Connection header received
0926 19:32:50.895082 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.895088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.895178 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.895282 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.896015 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55090: BlockingRecv error: recv got EOF from 127.0.0.1:55090 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:32:50.970342 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:50.963959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.1.39.61:40483)
0926 19:32:50.964204 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.964210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.964631 (+ 421us) server_negotiation.cc:407] Connection header received
0926 19:32:50.964852 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.964856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.964924 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.965043 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:50.965588 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.966409 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.968483 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:50.968979 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:50.969083 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:50.969626 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:50.969731 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:50.969810 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:50.969976 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 19:32:50.970045 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 19:32:50.978825 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:50.976678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55102 (local address 127.1.39.60:43629)
0926 19:32:50.976921 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:50.976927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:50.977287 (+ 360us) server_negotiation.cc:407] Connection header received
0926 19:32:50.977546 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:50.977551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:50.977641 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:50.977759 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:50.978621 (+ 862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55102: BlockingRecv error: recv got EOF from 127.0.0.1:55102 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 19:32:51.119714 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.114309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.1.39.61:40483)
0926 19:32:51.114550 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.114558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.114897 (+ 339us) server_negotiation.cc:407] Connection header received
0926 19:32:51.115127 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.115131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.115199 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.115301 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.115738 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.116460 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.118047 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.118483 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.118617 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.118997 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.119122 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.119219 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.119357 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.119438 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 19:32:51.127449 24217 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 19:32:51.125643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.1.39.60:43629)
0926 19:32:51.125864 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.125870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.126133 (+ 263us) server_negotiation.cc:407] Connection header received
0926 19:32:51.126333 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.126338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.126409 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.126515 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 19:32:51.127259 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55116: BlockingRecv error: recv got EOF from 127.0.0.1:55116 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 19:32:51.234679 24223 catalog_manager.cc:1259] Loaded cluster ID: b0ea64cad21a47a1bf5d9c2a368f747e
I20260926 19:32:51.234813 24223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: loading cluster ID for follower catalog manager: success
I20260926 19:32:51.237388 24223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: acquiring CA information for follower catalog manager: success
I20260926 19:32:51.239097 24223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a40c8c0415242099eb7b3ab64c3721b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 19:32:51.396016 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.390856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.1.39.61:40483)
0926 19:32:51.391092 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.391100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.391308 (+ 208us) server_negotiation.cc:407] Connection header received
0926 19:32:51.391564 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.391568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.391647 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.391749 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.392173 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.392904 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.394301 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.394724 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.394861 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.395251 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.395418 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.395517 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.395659 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.395738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 19:32:51.407410 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.401856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.1.39.60:43629)
0926 19:32:51.402116 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.402123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.402321 (+ 198us) server_negotiation.cc:407] Connection header received
0926 19:32:51.402512 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.402517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.402704 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.402809 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.403173 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.403865 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.405155 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.405621 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.405725 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.406229 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.406770 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.406883 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.407033 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.407102 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 19:32:51.424260 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.413455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.1.39.61:40483)
0926 19:32:51.413675 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.413684 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.417867 (+ 4183us) server_negotiation.cc:407] Connection header received
0926 19:32:51.418153 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.418162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.418276 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.418373 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.418997 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.420030 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.422491 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.422910 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.423048 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.423503 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.423652 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.423745 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.423854 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.423940 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 19:32:51.424969 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.417920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.1.39.60:43629)
0926 19:32:51.418133 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.418142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.418673 (+ 531us) server_negotiation.cc:407] Connection header received
0926 19:32:51.418904 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.418910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.419000 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.419068 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.419806 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.420797 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.422552 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.423187 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.423318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.423947 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.424076 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.424150 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.424607 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.424687 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 19:32:51.425060 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 19:32:51.413371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.1.39.62:38071)
0926 19:32:51.417835 (+ 4464us) server_negotiation.cc:206] Beginning negotiation
0926 19:32:51.417844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 19:32:51.417863 (+ 19us) server_negotiation.cc:407] Connection header received
0926 19:32:51.418222 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 19:32:51.418231 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 19:32:51.418491 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 19:32:51.418791 (+ 300us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 19:32:51.419256 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.420115 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.422539 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 19:32:51.423187 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 19:32:51.423318 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 19:32:51.423822 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 19:32:51.424223 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 19:32:51.424350 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 19:32:51.424746 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 19:32:51.424827 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4327,"thread_start_us":4258,"threads_started":1}
I20260926 19:32:51.429973 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 23962
I20260926 19:32:51.431006 24024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:51.555142 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 23962
W20260926 19:32:51.577179 24054 connection.cc:535] server connection from 127.0.0.1:58868 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 19:32:51.577788 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 24036
I20260926 19:32:51.578938 24098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:51.688812 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 24036
I20260926 19:32:51.709800 1180 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasksjShy7/build/asan/bin/kudu pid 24111
I20260926 19:32:51.710850 24173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 19:32:51.818439 1180 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksjShy7/build/asan/bin/kudu with pid 24111
2026-09-26T19:32:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4761 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9187 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (369853 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST